Research Logging

9. Research Logging#

9.1. Introduction#

One of the most valuable skills you can develop during your research experience is maintaining a thorough and consistent research log.

What is a research log? A research log (or lab notebook) is a detailed, chronological record of your research activities. It documents your hypotheses, experiments, observations, data, code, analysis steps, interpretations, new learnings, and even your thought processes.

Why is it crucial for you?

  • Reproducibility:- Allows you or others to replicate your work. This is a cornerstone of scientific research.

  • Memory Aid:- You’ll do and learn a lot! Your log will help you remember details weeks or months later.

  • Paper Writing:- When it’s time to write your final report or a publication, your log will be an invaluable source of information for the methods, results, and discussion sections.

  • Progress Tracking:- Helps you and your mentor see your progress and identify areas where you might be stuck.

  • Troubleshooting:- Documenting problems and how you solved them saves time if they recur.

  • Efficiency:- Prevents you from repeating mistakes or forgetting useful commands/settings.

Overview of Tools: We’ll focus on two excellent digital tools for research logging:

  • Jupyter Notebooks:- Ideal for integrating narrative text, live code (e.g., Python), equations (LaTeX), visualizations, and other rich media.

  • Markdown Files:- Lightweight, plain text files that are easy to write, version control, and convert to other formats.

Both can be effectively synced with cloud storage for backup and accessibility.

9.2. What Makes a Good Log Entry?#

While the exact format can be personalized, a good log entry is generally:

  • Dated:- Always start with the date of the entry. For any specific events, observations, or data timestamps recorded *within- the entry, always use Coordinated Universal Time (UTC). This is the standard in astronomy to avoid ambiguity.

  • Descriptive:- Clearly explain what you did, why, and what happened.

  • Organized:- Use headings, lists, and formatting to make it readable.

  • Detailed enough:- Include specifics like software versions, exact commands, code snippets, parameter settings, and filenames.

  • Honest:- Record what actually happened, including errors, unexpected results, and failed attempts. This is part of research!

Key Components of a Typical Daily/Task-Based Entry:

  • Date of Log Entry:- YYYY-MM-DD (This can be your local date when you are writing the log).

  • Title/Objective(s) for the day/task:- What did you plan to achieve?

  • Activities & Methodology:

    • What steps did you take?

    • What code did you write or run? (Include snippets)

    • What commands did you use? (e.g., terminal, LaTeX)

    • For observations: telescope, instrument, settings, conditions. All observation times must be in UTC.

  • Results & Observations:

    • What were the outcomes?

    • Include key figures, plots (can be embedded or referenced), or data summaries.

    • Any data timestamps (e.g., “Data received at YYYY-MM-DD HH:MM:SS UTC”) should be in UTC.

  • Problems Encountered & Solutions:

    • Any errors, bugs, or challenges?

    • How did you try to solve them? What worked?

  • New Learnings/Discoveries:

    • New commands, functions, concepts (Python, LaTeX, bash, software features).

    • Interesting insights from data.

  • Next Steps/To-Dos:

    • What’s planned for the next session?

    • Any pending questions for your mentor?

9.3. Method 1: Using Jupyter Notebooks for Your Research Log#

Jupyter Notebooks (.ipynb files) allow you to create and share documents that contain live code, equations, visualizations, and narrative text.

  • Pros:

    • Interactive code execution (Python, R, Julia, etc.) directly in your log.

    • Inline plots and images make results easy to see in context.

    • Rich text formatting using Markdown cells.

    • Can be a single, comprehensive file for an entry or a running log.

  • Cons:

    • Files can become large if they contain many high-resolution images or large datasets directly.

    • Version control diffs for .ipynb files can be less clean than for plain text, though tools are improving.

Setting Up:

  1. Organization:

    • Option A (Recommended for simplicity):- One notebook per day.

      • Filename: YYYY-MM-DD_Log.ipynb (e.g., 2025-06-01_Log.ipynb)

    • Option B:- One notebook per week, with daily entries separated by Markdown headers.

      • Filename: YYYY-WW_Log.ipynb (e.g., 2025-W22_Log.ipynb)

    • Option C:- One notebook per major project/task.

  2. Content:

    • Use *Markdown cells- for your narrative: objectives, descriptions, interpretations, learnings, etc.

    • Use *Code cells- for Python scripts, data analysis steps, generating plots, or even just to “store” command-line snippets (you can prefix shell commands with !).

Example Entry Structure within a Jupyter Notebook (using Markdown and Code cells):

(This example shows how you’d structure it using a mix of Markdown for text and showing code as text within Markdown, or as runnable code in a Code cell if you were actually demonstrating code execution related to logging, which is less common for the log’s narrative itself.)

9.3.1. Example Markdown Cell for a Log Entry:#

# Log Entry: 2025-06-01

## Objective(s)
- Familiarize myself with `lightkurve` for TESS data.
- Download and plot a light curve for star X.
- Record remote telescope settings for upcoming observation of star Y (scheduled for `2025-06-03 ~04:00 UTC`).

## Activities & Methodology

### 1. Lightkurve Exploration
Installed `lightkurve` using `pip install lightkurve`.
Worked through the basic tutorial on searching and downloading data.

**Code Snippet (Python example of what I ran):**
```python
# import lightkurve as lk
# search_result = lk.search_lightcurve("KIC 8462852", mission="Kepler")
# lc = search_result[0].download() # Data has time in BJD_TDB (UTC-based)
# lc.plot()
```
*(Self-note: The plot appeared inline showing the light curve.)*

### 2. Remote Telescope Settings Planning (for LCO 0.4m telescope)
Target: Star Y (RA: hh:mm:ss.s, Dec: dd:mm:ss.s)
Proposed Observation Window Start: `2025-06-03 03:30:00 UTC`
Filter sequence: V, B, R
Exposure times:
- V: 60s x 3
- B: 90s x 3
- R: 45s x 3
Binning: 1x1
Readout Speed: Standard
Focus offset: Check recent autofocus, otherwise 0.

## Results & Observations
- Successfully plotted Kepler light curve for KIC 8462852. Timestamps in BJD are UTC-based.
- Telescope settings for Star Y documented for submission to LCO portal.

## Problems Encountered & Solutions
- Initially forgot to import `matplotlib.pyplot` for a custom plot tweak, got `NameError`. Added `import matplotlib.pyplot as plt` in my analysis script.

## New Learnings
- **Python/Lightkurve:*- `lk.search_lightcurve()` returns a `SearchResult` object. `lc.download()` fetches the data. `lc.plot()` is a quick way to visualize. **Kepler/TESS times are typically in BJD or BKJD, which are UTC-based.**
- **Remote Observing:*- Realized I need to check the airmass and moon phase for Star Y for the planned `~03:30 UTC` start time.

## Next Steps
- Submit observation request for Star Y (target window `2025-06-03 03:30-05:00 UTC`).
- Explore `lightkurve` functions for cleaning TESS light curves.

9.4. Method 2: Using Markdown Files for Your Research Log#

Markdown (.md files) is a lightweight markup language with plain-text formatting syntax. It’s widely used for README files, documentation, and note-taking.

  • Pros:

    • Very lightweight and fast to open/edit.

    • Plain text makes it excellent for version control systems like Git (easy to see changes).

    • Highly portable – can be opened by almost any text editor.

    • Many editors (like VS Code, Typora, Obsidian) offer live previews.

  • Cons:

    • Code is not directly executable within the log file itself (it’s just text).

    • Plots and images are typically external files linked in the Markdown (though some advanced Markdown flavors/editors can embed).

Setting Up:

  1. Organization:- Similar to Jupyter Notebooks, create files per day, week, or task.

    • Filename: YYYY-MM-DD_Log.md or ProjectX_Log.md

  2. Editor:- Use any plain text editor. VS Code is highly recommended as it has excellent Markdown support (syntax highlighting, live preview).

  3. Markdown Syntax:- Familiarize yourself with basic Markdown:

    • # Heading 1, ## Heading 2

    • *italic* or _italic_, **bold** or __bold__

    • - Unordered list item or - Unordered list item

    • 1. Ordered list item

    • `inline code`

    • # Code block
x = 10
print(x)

    • [Link text](URL)

    • ![Alt text for image](path/to/image.png)

Example Entry Structure within a Markdown File: The content would be identical to the Jupyter Notebook example’s Markdown cell, typed directly into the .md file. For plots, you might save the plot as an image file (e.g., starX_lightcurve.png) and then link it:

![Light curve of Star X](figures/2025-06-01_starX_lightcurve.png)

(Ensure any time axes on plots are also clearly labeled, preferably in a UTC-based system like JD or BJD).

9.5. Content Spotlight: What to Record#

Be specific! Future You will thank you. Remember to use UTC for all event and data timestamps.

9.5.1. Daily Learnings:#

  • Python/Packages (e.g., Matplotlib, NumPy, Astropy, Pandas, Lightkurve):

    • “Learned plt.xlabel('Time (BJD_TDB)') to add x-axis label. BJD_TDB is UTC-based.”

    • “Used numpy.loadtxt('data.txt', skiprows=1) to load data. Noted that timestamps in file are MJD (UTC).”

  • LaTeX:

    • “Found \\usepackage{amsmath} is needed for \\align environment.”

  • Unix/Terminal Commands:

    • “Used ls -lrt to list files by modification time, newest last.”

    • “Transferred file from remote server: scp myuser@remote.edu:/path/to/data.fits . File creation time on server was YYYY-MM-DD HH:MM:SS UTC.”

9.5.2. Remote Telescope Observations:#

This is critical for reproducibility and understanding your data. All times MUST be in UTC.

  • Date & Time of Observation (UTC Start to End):- e.g., 2025-06-15 03:45:00 UTC to 2025-06-15 04:30:00 UTC

  • Telescope Used:- e.g., LCO 0.4m Faulkes Telescope North (fl03)

  • Target Name & Coordinates:- NGC 6946, RA: 20:34:52.3, Dec: +60:09:14

  • Proposal ID / Program Name:- (If applicable)

  • Instrument Settings:

    • Filter(s) used:- Sloan g', Sloan r', Sloan i'

    • Exposure Time(s):- g': 120s, r': 90s, i': 90s

    • Number of Exposures per filter:- 3 each

    • Binning:- 1x1 (or 2x2)

    • Readout Mode/Speed:- Standard / 1MHz

  • Observing Conditions (if known/logged by system):

    • Seeing:- 1.5 arcsec FWHM (noted at ~04:00 UTC)

    • Airmass at start/end:- 1.2 -> 1.1

  • Software Used for Request/Observation:- LCO Portal, TOM Toolkit vX.Y

  • Commands/Scripts Used:- Submitted via LCO API script: observe_ngc6946.py on 2025-06-14 22:15:00 UTC

  • Raw Data Filenames & Storage Location (on server/archive):

    • e.g., fl03.20250615.0034.fits, ... (FITS headers will contain UTC timestamps)

    • Downloaded to /project/data/raw/YYYY-MM-DD/ at 2025-06-15 08:00:00 UTC

  • Calibration Frames Used/Requested:- Standard set of bias, darks, flats for night of 2025-06-15 (UTC).

9.5.3. Code Snippets & Outputs:#

  • Don’t paste huge scripts, but include the relevant function or loop.

  • Describe key outputs or embed/link plots. “The resulting scatter plot (see scatter_correlation.png) showed a positive trend. Data points timestamped in BJD_TDB.”

9.5.4. Problems & Solutions:#

  • “My Python script kept crashing with MemoryError. Realized I was loading too many full FITS images at once. Modified to process them one by one. This occurred while processing data from 2025-06-10 UTC.”

  • “Remote telescope observation on 2025-06-12 UTC failed due to ‘target too close to meridian limit’. Rescheduled for 2025-06-13 05:00 UTC.”

9.6. Syncing Your Research Log with the Cloud#

Keeping your log synced to the cloud is vital for backup, accessibility from multiple devices, and easy sharing with your mentor.

9.6.1. Option A: Google Colab and Google Drive (primarily for Jupyter Notebooks)#

Google Colaboratory (Colab) is a free Jupyter Notebook environment that runs in the cloud and integrates seamlessly with Google Drive.

  1. Working in Colab:

    • Go to colab.research.google.com.

    • You can create new notebooks (File > New notebook) or open existing ones from Google Drive, GitHub, or by uploading.

  2. Automatic Saving to Google Drive:

    • By default, Colab notebooks you create or modify are saved to a folder called Colab Notebooks in your Google Drive.

  3. Organization:

    • Create a dedicated folder in your Google Drive for your project (e.g., Summer_2025).

    • Inside that, create a subfolder for your research log (e.g., Research_Log).

    • Move your log notebooks (YYYY-MM-DD_Log.ipynb) into this folder.

  4. Accessibility:- Access your log from any computer with internet access by logging into your Google account.

  5. Sharing:- You can easily share individual notebooks or the entire folder with your mentor for feedback using Google Drive’s sharing options (granting “Viewer” or “Commenter” access is usually best for logs).

9.6.2. Option B: GitHub (for Jupyter Notebooks or Markdown files)#

GitHub is a platform for version control and collaboration using Git. It’s excellent for tracking changes, reverting to previous versions, and collaborating.

  1. Why GitHub for a log?

    • Version Control:- Every commit is a snapshot. You can see how your work evolved and revert changes if needed.

    • Backup:- Your log is stored remotely.

    • Industry Standard:- Familiarity with Git/GitHub is a valuable skill.

  2. Setup:

    • Create a GitHub account if you don’t have one.

    • Create a new *private repository- for your research log (e.g., research-log). A private repo ensures only you (and anyone you explicitly invite) can see it.

  3. Basic Git Workflow (from your computer’s terminal/Git client):

    • Clone the repository (do this once):

      git clone [https://github.com/YourUsername/research-log.git](https://github.com/YourUsername/research-log.git)
cd research-log

    • Create/Add your log file: Place your YYYY-MM-DD_Log.ipynb or YYYY-MM-DD_Log.md file into this cloned directory.

    • Stage your changes (add files to be committed):

      git add YYYY-MM-DD_Log.ipynb  # Add a specific file
# OR
git add .                     # Add all new/modified files in the current directory

    • Commit your changes (save a snapshot): Write a meaningful commit message.

      git commit -m "Log entry for YYYY-MM-DD: explored lightkurve, planned telescope settings (UTC)"
# For longer messages, just `git commit` will open a text editor

    • Push your changes to GitHub (upload):

      git push origin main   # Or 'master' if that's your default branch name

  4. Frequency:- Aim to add, commit, and push your log at the end of each day or significant work session.

  5. Viewing History:- You can see your commit history on your GitHub repository page.

  6. .gitignore (Optional but good practice for Jupyter Notebooks): Jupyter Notebooks create checkpoint files. You might not want to track these. Create a file named .gitignore in your repository’s root directory with the following content:

    # Jupyter Notebook checkpoints
.ipynb_checkpoints/
__pycache__/
*.pyc

    Commit this .gitignore file to your repository.

9.7. Tips for Maintaining an Effective Log#

  • Be Consistent:- Make it a daily habit, even if it’s just a few notes. It’s harder to reconstruct things later.

  • Write for Your Future Self:- Imagine you’re reading this log six months or a year from now. Will it make sense? Provide enough context.

  • Don’t Strive for Perfection:- It’s a log, not a polished paper. Clarity and completeness are more important than beautiful prose.

  • Record Failures Too:- “Attempted X, but it failed because of Y. Realized Z.” This is valuable learning!

  • Use Templates:- If you find yourself writing the same sections every day, create a simple template in a blank notebook or Markdown file and copy it for new entries.

  • Cross-Reference:- If an entry builds on a previous one, link to it or mention the date.

  • Use UTC for Time-Sensitive Information:- Reinforce this habit for all astronomical data, observations, and event logging.

9.8. 8. Conclusion#

Your research log is more than just a requirement; it’s one of your most powerful research tools. A well-maintained log will significantly reduce stress when writing reports, help you learn more effectively, and provide a valuable record of your accomplishments.

Choose the tool (Jupyter Notebook or Markdown) and syncing method (Google Drive or GitHub) that works best for you, and commit to using it regularly. Your future self will be grateful!

Contents