Intensities incorrect for ST-VIS-25

spectrometer and system information

model: ST, Serial: ST02185, ST-VIS-25
operating system: Jetson Orin Ubuntu 22.04 - ARM processor
python version: Python 3.10.12
python-seabreeze version: current master
installed-via: pip install seabreeze

current problem

When I try to use the script Seabreeze script I get a different array size of intensities array than when I do with the OceanDirect script. I'm not sure if the corrections are applied either? We want to run on an ARM processor

Seabreeze: First 10 values: [1505. 0. 1563. 0. 1570. 0. 1544. 0. 1522. 0.]
Size: 3032

OceanDirect: First 10 values: [14.654972076416016, 14.654972076416016, 13.45759105682373, -2.121690034866333, -0.9224371910095215, 4.272670745849609, 5.471170902252197, -2.521484851837158, 5.471170902252197, 3.8731637001037598]
Size: 1516

steps to reproduce

(venv) ~/seabreeze$ python3
Python 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import seabreeze
>>> seabreeze.use('pyseabreeze')
>>> from seabreeze.spectrometers import Spectrometer
>>> spec = Spectrometer.from_first_available()
>>> print(f"Connected to spectrometer: {spec.model}, Serial: {spec.serial_number}")
Connected to spectrometer: ST, Serial: ST02185
>>> spec.integration_time_micros(100000)
>>> spec.wavelengths()
array([348.31332397, 348.65684909, 349.00031106, ..., 811.36221513,
       811.64102486, 811.91981221])
>>> print(f"Wavelengths shape: {spec.wavelengths().shape}")
Wavelengths shape: (1516,)
print(f"First 10 values: {spec.wavelengths()[:10]}")
First 10 values: [348.31332397 348.65684909 349.00031106 349.3437099  349.68704565
 350.03031833 350.37352796 350.71667458 351.05975821 351.40277888]
>>> spec.intensities()
array([1515.,    0., 1527., ...,    0., 1524.,    0.])
>>> print(f"Intensities shape: {spec.intensities().shape}")
Intensities shape: (3032,)
>>> print(f"First 10 values: {spec.intensities()[:10]}")
First 10 values: [1505.    0. 1563.    0. 1570.    0. 1544.    0. 1522.    0.]

When I run a similar script using the the OceanDirect script I get these response from the spectrometer:

Connected to spectrometer: OBP2Device, Serial: ST02185
Dark Pixel Information:
Number of electric dark pixels: 13
Dark pixel indices: [1503, 1504, 1505, 1506, 1507, 1508, 1509, 1510, 1511, 1512, 1513, 1514, 1515]

Raw spectrum values at dark pixel locations:
Index 1503: 33.334922790527344
Index 1504: 5.809284210205078
Index 1505: 5.809284210205078
Index 1506: 17.7861270904541
Index 1507: -18.186737060546875
Index 1508: -20.58944320678711
Index 1509: -3.78235125541687
Index 1510: -9.781708717346191
Index 1511: -50.67084884643555
Index 1512: -74.79936981201172
Index 1513: -27.80094337463379
Index 1514: 45.27930450439453
Index 1515: 96.47938537597656

OceanDirect Formatted Spectrum:
Length: 1516
First 10 values: [50.05310821533203, 50.05310821533203, 81.02778625488281, 45.27930450439453, -2.5829029083251953, -4.981940746307373, 45.27930450439453, -6.181671142578125, -20.58944320678711, -6.181671142578125]
INFO: Buffer size returned: 4

Calibration Coefficients:
Nonlinearity coefficients: [0.8339769840240479, 4.907249967800453e-05, 3.217550004208647e-09, -3.677040009281152e-12, 6.984339868505382e-16, -6.182079710757103e-20, 2.6929200754414477e-24, -4.671669886348679e-29]
Wavelength coefficients: [348.3133239746094, 0.3435567021369934, -3.158873732900247e-05, 4.492421545876368e-09]
Integration time set to 0.1 seconds.
Scans to average set to 3.

Wavelengths array details:
Size: 1516
First 10 values: [348.3133239746094, 348.6568603515625, 349.00030517578125, 349.3437194824219, 349.6870422363281, 350.0303039550781, 350.37353515625, 350.7166748046875, 351.05975341796875, 351.40277099609375]
Last 10 values: [809.409912109375, 809.6889038085938, 809.9678344726562, 810.2467651367188, 810.525634765625, 810.8045654296875, 811.0833740234375, 811.3621826171875, 811.6410522460938, 811.9197998046875]

Raw spectrum array details:
Size: 1516
First 10 values: [14.654972076416016, 14.654972076416016, 13.45759105682373, -2.121690034866333, -0.9224371910095215, 4.272670745849609, 5.471170902252197, -2.521484851837158, 5.471170902252197, 3.8731637001037598]
Last 10 values: [15.054130554199219, 14.654972076416016, 0.27667462825775146, -0.522741973400116, -10.120340347290039, -20.127416610717773, -0.9224371910095215, 13.45759105682373, 1.4756454229354858, -5.720295429229736]

minimal code example and error (very helpful if available)

import seabreeze
seabreeze.use("pyseabreeze")  # Ensure the pyseabreeze backend is used
from seabreeze.spectrometers import Spectrometer, list_devices
from datetime import datetime
import csv
import matplotlib.pyplot as plt
import pytz
import numpy as np

# Configuration Variables
INTEGRATION_TIME_MICROS = 100000  # Integration time in microseconds (e.g., 100000 = 100 ms)
SCANS_TO_AVERAGE = 3  # Number of scans to average
TIMEZONE = "Australia/Brisbane"  # Set your desired timezone or "UTC"

def capture_spectrum_and_plot():
    try:
        # Initialize the spectrometer
        spec = Spectrometer.from_first_available()
        print(f"Connected to spectrometer: {spec.model}, Serial: {spec.serial_number}")
        
        # Set integration time
        spec.integration_time_micros(INTEGRATION_TIME_MICROS)
        print(f"Integration time set to {INTEGRATION_TIME_MICROS / 1e6} seconds.")
        
        # Retrieve wavelengths and average multiple spectra
        wavelengths = spec.wavelengths()
        intensities_list = []
        print(f"Wavelengths shape: {wavelengths.shape}")
        print(spec.wavelengths())
        for _ in range(SCANS_TO_AVERAGE):
            intensities_list.append(spec.intensities(correct_dark_counts=True, correct_nonlinearity=True))
        intensities = np.mean(intensities_list, axis=0)
        print(f"Intensities shape: {intensities.shape}")
        print(spec.intensities())
        
        # Get timestamp with timezone
        tz = pytz.timezone(TIMEZONE)
        now = datetime.now(tz)
        timestamp = now.isoformat()
        filename_timestamp = now.strftime("%Y%m%d%H%M%S")
        output_csv = f"spectrum_{filename_timestamp}.csv"
        output_png = f"spectrum_{filename_timestamp}.png"
        
        # Save data to CSV
        with open(output_csv, mode="w", newline="") as file:
            writer = csv.writer(file)
            writer.writerow(["Timestamp", timestamp])
            writer.writerow(["Model", spec.model])
            writer.writerow(["Serial Number", spec.serial_number])
            writer.writerow(["Integration Time (s)", INTEGRATION_TIME_MICROS / 1e6])
            writer.writerow(["Scans Averaged", SCANS_TO_AVERAGE])
            writer.writerow([])
            writer.writerow(["Wavelength (nm)"] + list(wavelengths))
            writer.writerow(["Intensity"] + list(intensities))
        print(f"Spectrum data saved to {output_csv}")
        
        # Create and save plot
        plt.figure(figsize=(10, 6))
        plt.plot(wavelengths, intensities, label="Spectrum", color="blue", linewidth=1.5)
        plt.title(f"Spectrum - {timestamp}", fontsize=14)
        plt.xlabel("Wavelength (nm)", fontsize=12)
        plt.ylabel("Intensity", fontsize=12)
        plt.grid(True, linestyle="--", alpha=0.6)
        plt.legend(loc="upper right")
        plt.tight_layout()
        plt.savefig(output_png, dpi=300)
        print(f"Spectrum plot saved to {output_png}")
        
        # Close the spectrometer connection
        spec.close()
        
    except Exception as e:
        print(f"Error: {e}")

# Run the function
if __name__ == "__main__":
    capture_spectrum_and_plot()

Changing to a 32-bit signed integer and increasing the integration time from 100000 to 1000000 got a similar plot to OceanDirect:

class SeaBreezeSpectrometerFeatureOBP2(SeaBreezeSpectrometerFeatureOBP):
    def get_intensities(self) -> NDArray[np.float_]:
        tmp = self._get_spectrum_raw()
        # Skip the 32-byte metadata header
        ret = tmp[32:].view(numpy.dtype("<i4")).astype(numpy.double)
        return ret * self._normalization_value

Seabreeze:

OceanDirect:

Thanks for reporting!

Would you be interested in contributing support for the ST-VIS-25?

The complicated part here is seemingly (and I am guessing here) that all the ST models share the same USB PID, so we need a similar mechanism as used for the USB2000+ and FlameS models where intitially the default model class is used but then replaced with the correct model dependent on the detected ST model.

If you'd like to work on a PR, I would write a step by step guide to help implement support.

Cheers,
Andreas

Yeah, I would love to help.

Scans to average error

I think I may have identified what was causing the initial error. The ocean direct set_scans_to_average function changes the raw intensities output if set_scans_to_average is greater than 1:

For example:
SCANS_TO_AVERAGE = 2 # Number of scans to average greater than 1
device.set_scans_to_average(SCANS_TO_AVERAGE)

Then run the pyseabreeze script, the intensities array appears to double with every second value as a zero, and the intensities appear to be a multiple of whatever I had SCANS_TO_AVERAGE set to. If I rerun the set_scans_to_average set to 1, the array goes back to the default length and no error.

For example when I set to 1, both scripts work as expected:

SCANS_TO_AVERAGE = 1 # In Ocean Direct script
Ocean Direct output:
Intensities array length: 1516
First 10 raw intensities values: [523.5, 523.5, 513.0, 510.5, 522.5, 512.5, 515.5, 510.0, 499.0, 522.0]

Pyseabreeze output:
Intensities array length: 1516
First 10 raw intensities values: [504.0, 504.0, 528.0, 531.0, 534.0, 534.0, 527.0, 525.0, 510.0, 523.0]
First 10 dark values: [-5.23077392578125, -5.23077392578125, -19.23077392578125, -29.23077392578125, -8.23077392578125, 19.76922607421875, 22.76922607421875, 11.76922607421875, -12.23077392578125, -3.23077392578125]

Setting the set_scans_to_average to a value greater than 1 causes the pyseabreeze error:

Ocean Direct output:
SCANS_TO_AVERAGE = 2 # In Ocean Direct script
Intensities array length: 1516
First 10 raw intensities values: [523.5, 523.5, 513.0, 510.5, 522.5, 512.5, 515.5, 510.0, 499.0, 522.0]
First 10 dark values: [2.30767822265625, 2.30767822265625, 8.80767822265625, 6.80767822265625, 19.30767822265625, 18.30767822265625, -12.19232177734375, -20.69232177734375, -23.69232177734375, -10.69232177734375]

Pyseabreeze output:
Intensities array length: 3032
First 10 raw intensities values: [1037. 0. 995. 0. 996. 0. 1032. 0. 1004. 0.]
First 10 dark values: [ 401.15384615 -658.84615385 419.15384615 -658.84615385 405.15384615 -658.84615385 393.15384615 -658.84615385 373.15384615 -658.84615385]

Ocean Direct output:
SCANS_TO_AVERAGE = 3 # In Ocean Direct script

Pyseabreeze output:
Intensities array length: 3032
First 10 raw intensities values: [1579. 0. 1564. 0. 1548. 0. 1588. 0. 1589. 0.]
First 10 dark values: [ 477.46153846 -1054.53846154 466.46153846 -1054.53846154 447.46153846 -1054.53846154 468.46153846 -1054.53846154 496.46153846 -1054.53846154]

This wouldn't be an issue in normal operation. I guess I only encountered this because I wanted to validate the pyseabreeze results where in line with ocean direct.

Integration time discrepancy

A second issue also seems to be the integration time when I compare the two outputs?

If I set the pyseabreeze integration time to the same time as Ocean Direct (100000usec):

If I up the pyseabreeze integration time to (1000000usec) the results look very similar:

PYSEABREEZE_INTEGRATION_TIME_MICROS = 1000000
OCEANDIRECT_INTEGRATION_TIME_MICROS = 100000

If I set both to (1000000usec):

PYSEABREEZE_INTEGRATION_TIME_MICROS = 1000000
OCEANDIRECT_INTEGRATION_TIME_MICROS = 1000000

I think the scale is wrong in pyseabreeze for the ST? Increasing the integration_time_max = 60000000 so I can increase the pyseabreeze integration time to 10000000 looks the same as 1000000 for the Ocean Directs script (note: axis scale is a little different in this image).

Non-linearity result difference

One other thing I noticed in the script above. Ocean Directs non-linearity correction appears to be corrected for dark pixels even when off as it's shifted?

Ocean Direct script:

Nonlinearity-corrected only

    device.set_electric_dark_correction_usage(False)
    device.set_nonlinearity_correction_usage(True)
    non_linearity_corrected = device.get_formatted_spectrum()

Versus the pyseabreeze script:

non_linearity_corrected = spec.intensities(correct_dark_counts=False, correct_nonlinearity=True)

Navigation Menu

Repository navigation

Intensities incorrect for ST-VIS-25 #272

spectrometer and system information

current problem

steps to reproduce

minimal code example and error (very helpful if available)

Scans to average error

Integration time discrepancy

Non-linearity result difference

Nonlinearity-corrected only

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Intensities incorrect for ST-VIS-25 #272

Description

spectrometer and system information

current problem

steps to reproduce

minimal code example and error (very helpful if available)

Activity

roboticsmick commented on Nov 26, 2024

ap-- commented on Dec 10, 2024

roboticsmick commented on Dec 11, 2024

Scans to average error

Integration time discrepancy

Non-linearity result difference

Nonlinearity-corrected only

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions