EC&V Pty Ltd: TROPOMI CSF: Commands

TROPOMI CSF Commands

Data for this project and analysis comes from different public source. The code was written in the first half of 2025 and many data providers modified their interfaces so we do not give any warranties that this code will work in the future. For example, BoM updated its web site in November 2025 and although API URLs are currently valid these are under BoM control.

To configure your project please update “Config.py” file. It defines location of data, domains, and algorithm parameters.

The full list of programs is available in the provided “Readme.md” file.

To obtain help for each program please use “-h” option. For example, to obtain help for program “Copernicus_Catalogue.py” please use:

python .\Copernicus_Catalogue.py -h

As this software deals with TROPOMI satellite images. Obtaining the TROPOMI image is the first step in analysis and most of the remaining data sets will depend on this image.

TROPOMI - Copernicus Data Space Ecosystem

“Copernicus_Catalogue.py” package contains utilities for querying orbits available at Copernicus Data Space Ecosystem . “Copernicus_S3.py” package contains utility for downloading selected file from S3 storage managed by Copernicus.

To access TROPOMI data, you will need to create account on Copernicus Data Space Ecosystem web site. These credentials will allow you to search through the Copernicus Data Space catalogue and also generate S3 credentials to download TROPOMI files. The procedure for creating credentials is described in the Copernicus Data Space S3 web page. After credentials are created a window opens allowing you copy “access_key” and “access_secret”, you need to copy these as these are visible only once.

To search for orbits containing domain, for example on 15 September 2019, use the following command, replacing username and password with your account details:

python .\Copernicus_Catalogue.py 20190915 username password

The expected output (as of December 2025) lists S3_keys for two data sets available:

Sentinel-5P/TROPOMI/L2__CH4___/2019/09/15/S5P_RPRO_L2__CH4____20190915T030716_20190915T044845_09956_03_020400_20221114T145812.nc
Sentinel-5P/TROPOMI/L2__CH4___/2019/09/15/S5P_OFFL_L2__CH4____20190915T030716_20190915T044845_09956_01_010302_20190921T050748.nc

To download specific TROPOMI orbit from S3 use the command:

python .\Copernicus_S3.py download single access_key access_secret S3_key

with the following parameters:

access_key - S3 access key id obtained from Copernicus Data Store
access_secret - S3 access key secret obtained from Copernicus Data Store
S3_key - S3 key returned by the above Copernicus_Catalogue command,

TROPOMI files can also be downloaded manually from the Australian Hub of Copernicus Data Store which is maintained by Geoscience Australia Copernicus Australasia Hub . It should be noted though that at the time of writing this report, data series start from January 1 ^st 2025 in this repository and older files need to be downloaded from S3 storage.

To download all files processed by TROPOMI_Filter “contains” filter the following command should be used:

python .\Copernicus_S3.py download filter access_key access_secret

HINT: Please note that this may download some Terabytes of data

TROPOMI_Filter

“TROPOMI_Filter.py” package contains a set of utilities to carry the initial filtering of TROPOMI files. It contains 3 sequential filters: “contains”, “qa” and “r”. The first filter uses CDSE search function to select only orbits which contain the domain specified in the Config file. To run “qa” and “r” filters you will need to download required files locally first.

Please note that filters depend on previous filter that is “qa” needs to be run after “contains” and “r” after “qa”.

To run the “contains” filter the following command should be used:

python .\TROPOMI_Filter.py run contains username password

HINT: To run the remaining filters, you will need to download required files locally.

To list output of the filter the following command should be used:

python .\TROPOMI_Filter.py list contains

An abbreviated output will look like:

Filter contains
        First date inclusive: 2018-04-30 00:00:00+00:00 Last date exclusive: 2020-01-01 00:00:00+00:00
        Domain bounds: [146.0, -24.0, 150.0, -20.0]
        Last date processed: 2019-12-31
        Filtered count: 757
        Filtered orbits:
[2820, 2834, 2848, 2862, 2877,
…
11431, 11446, 11460, 11474]
Filter completed: True

TROPOMI

Once you have access to TROPOMI file you can use TROPOMI.py package to display TROPOMI images. The following commands may be useful in analysis. To display an image corresponding to TROPOMI image for orbit 09956, processor 010302 box with left bottom corner of (135°E, 35°S) and right upper corner (155°E, 15°S).

python .\TROPOMI.py chart box HailCreek 09956 010302 135.0 -35.0 155.0 -15.0

Another chart will draw the image corresponding to the domain of the experiment.

python .\TROPOMI.py chart domain HailCreek 09956 010302

The final version draws a box of specified size around Hail Creek mine where the last parameter describes distance in degrees away from the source (that is when size is 1.0, it will draw a box of 2 degrees in longitude and latitude)

python .\TROPOMI.py chart size HailCreek 09956 010302 1.0

ERA5 - Copernicus Data Space Ecosystem

ERA5.py package contains utilities for reading and analysing ERA5 files. ERA5 package depends on TROPOMI file in order to calculate the closest grid time to that of the satellite passing over Hail Creek mine.

To access ERA5 data, you will need an account on Copernicus Data Store web site. Due to requirement of cdsapi package, these credentials need to be placed in the file called “.cdsapirc” as you cannot use them as command parameters. For documentation of this file please check Copernicus documentation.

To download ERA5 required for CSF calculation at Hail Creek for orbit 09956, processor 010302 use the following command:

python .\ERA5.py download CSF HailCreek 09956 010302

To list data used to calculate wind you can use the command:

python .\ERA5.py list CSF HailCreek 09956 010302

This will display the output similar to:

TROPOMI orbit: 09956
        scan: 1509 pixel: 105 nearest hour: 2019-09-15:04Z
Single level variables
        Boundary Layer Height: 2269.34228515625 (m), Surface Pressure: 977.3350219726562 (hPa)
Pressure level variables
        Pressure: 1000.0 (hPa), Height: 166.2404327392578 (m), u: -3.9813385009765625 (m/s), v: -0.6917724609375 (m/s)
        Pressure: 975.0 (hPa), Height: 384.97369384765625 (m), u: -4.2635040283203125 (m/s), v: -0.7312164306640625 (m/s)
        Pressure: 950.0 (hPa), Height: 612.2296142578125 (m), u: -5.9913787841796875 (m/s), v: -1.1244049072265625 (m/s)
        Pressure: 925.0 (hPa), Height: 843.7330932617188 (m), u: -6.1446533203125 (m/s), v: -1.164459228515625 (m/s)
        Pressure: 900.0 (hPa), Height: 1079.68310546875 (m), u: -6.1470947265625 (m/s), v: -1.1403961181640625 (m/s)
        Pressure: 875.0 (hPa), Height: 1320.3408203125 (m), u: -6.0863189697265625 (m/s), v: -1.09808349609375 (m/s)
        Pressure: 850.0 (hPa), Height: 1565.921142578125 (m), u: -5.959503173828125 (m/s), v: -1.1395263671875 (m/s)
        Pressure: 825.0 (hPa), Height: 1816.726806640625 (m), u: -5.6634521484375 (m/s), v: -1.1598358154296875 (m/s)
        Pressure: 800.0 (hPa), Height: 2072.953125 (m), u: -5.3851776123046875 (m/s), v: -1.03070068359375 (m/s)
        Pressure: 775.0 (hPa), Height: 2335.0283203125 (m), u: -4.967010498046875 (m/s), v: -0.6363372802734375 (m/s)
        ...
        Pressure: 1.0 (hPa), Height: 47505.69140625 (m), u: 31.1484375 (m/s), v: -4.4208984375 (m/s)

To list pressure averaged PBL wind you can use the command:

python .\ERA5.py wind CSF HailCreek 09956 010302

This will generate an output that looks like:

BLH pressure averaged wind. u: -5.6952834868095294 (m/s), v: -1.0697813437018595 (m/s), speed: 5.794884478439399 (m/s)

MSLP - BOM

To download BOM MSLP image run the following command in your project:

python .\BOM_MSLP.py download HailCreek 09956 010302

You can display a chart using the following command:

python .\BOM_MSLP.py chart HailCreek 09956 010302

AWS - BOM

“BOM_AWS.py” package contains utilities for analysis of the AWS data. It depends on Moranbah AWS data which need to be purchased from the BoM as a CSV file in fixed format described in this report.

The following command will display a single observation at 201902111000:

python .\BOM_AWS.py list Moranbah 201902111000 201902111000

This will display the output similar to:

Temperature Air: 30.7 (C)
Temperature Wet Bulb: 18.4 (C)
Dew Point: 9.0 (C)
Relative Humidity: 26 (%)
Precipitation 10min: 0.0 (mm)
Precipitation since 9am (Local): 0.0 (mm)
Cloud amount (group 1):
Cloud amount (group 2):
Cloud amount (group 3):
Wind speed: 7.6 (km/h)
Wind direction: 280 (True deg)
Max wind gust: 9.4 (km/h)
Visibility: 10.0 (km)
Pressure Mean Sea Level: 1003.3 (hPa)
Pressure Station Level: 977.4 (hPa)
Pressure QNH: 1004.9 (hPa)

To display a list of observation between 201902111000 and 201902111200:

python .\BOM_AWS.py list Moranbah 201902111000 201902111200

This will display the output similar to:

2019-02-11 10:00 (UTC),30.7 (C),18.4 (C),9.0 (C),26 (%),0.0 (mm),0.0 (mm),7.6 (km/h),280 (True deg),9.4 (km/h),,,,,,,10.0 (km),1003.3 (hPa),977.4 (hPa),1004.9 (hPa)
2019-02-11 10:30 (UTC),27.2 (C),17.8 (C),10.8 (C),36 (%),0.0 (mm),0.0 (mm),3.6 (km/h),240 (True deg),13.0 (km/h),,,,,,,10.0 (km),1003.8 (hPa),977.9 (hPa),1005.4 (hPa)
2019-02-11 11:00 (UTC),30.6 (C),23.0 (C),19.0 (C),50 (%),0.0 (mm),0.0 (mm),16.6 (km/h),30 (True deg),24.1 (km/h),,,,,,,10.0 (km),1004.3 (hPa),978.4 (hPa),1005.9 (hPa)
2019-02-11 11:30 (UTC),28.9 (C),22.9 (C),19.8 (C),58 (%),0.0 (mm),0.0 (mm),13.0 (km/h),50 (True deg),14.8 (km/h),,,,,,,10.0 (km),1004.5 (hPa),978.6 (hPa),1006.1 (hPa)
2019-02-11 12:00 (UTC),28.8 (C),21.3 (C),17.0 (C),49 (%),0.0 (mm),0.0 (mm),7.6 (km/h),110 (True deg),13.0 (km/h),,,,,,,10.0 (km),1004.8 (hPa),978.9 (hPa),1006.4 (hPa)

This package provides also a utility to display a list of observation preceding specific point in time. So, to list all observations within 24 hours before 201902111000 use:

python .\BOM_AWS.py list_period Moranbah 201902111000 24

Finally, to display windrose chart for all observations in this file use:

python .\BOM_AWS.py chart windrose

Source

“Source.py” package contains information about mines. This package contains activity data for Qld mines active in FY 2019, 2020 as provided by Queensland Government. Unfortunately, location data are rather crude based on Google queries.

Data coded for Hail Creek can be listed using the following command:

python .\Source.py list HailCreek

This will generate an output that looks like:

Code name: HailCreek Display name: Hail Creek      
Longitude: 148.3631389, Latitude: -21.490934       
ERA5 grid point: Longitude: 148.25, Latitude: -21.5
Activity data:
        2025: 10.807512
        2024: 9.592586
        2023: 10.092999
        2022: 10.783931
        2021: 10.594483
        2020: 9.468696
        2019: 7.660496
        2018: 10.203613
        2017: 9.195752
        2016: 10.228814
        2015: 11.614548
        2014: 11.911568
        2013: 12.06333
        2012: 12.817071
        2011: 11.983327
        2010: 12.554146

HYSPLIT

“Algorithm_HYSPLIT.py” package provides utilities for combining TROPOMI files and HYSPLIT modelling. It depends on TROPOMI files and downloaded HYSPLIT trajectories.

The following command will create empty files for backward and forward HYSPLIT trajectories starting at the time of the orbit 09942 and ending at the time of the image 09956:

python .\Algorithm_HYSPLIT.py create HailCreek 09956 010302 09942 010302 GFSQ

You will need to run HYSPLIT from its web site to populate these files.

To chart forward trajectories passing over Hail Creek at the time of satellite pass for orbit 09556, processor 09956 and overlaying over the same image you can use the following command (entered as a single line):

python .\Algorithm_HYSPLIT.py chart slice HailCreek 09956 010302 09956 010302 GFSQ F 2019091404 2019091503 2019091504 2019091504 2019091504

Similarly to chart backward trajectories overlaid over orbit 09942 use the following command:

python .\Algorithm_HYSPLIT.py chart slice HailCreek 09956 010302 09942 010302 GFSQ B 2019091404 2019091404 2019091405 2019091503 2019091404

To chart a single air parcel trajectory starting at 2019091404, passing over Hail Creek at 2019091500 and terminating at 2019091504:

python .\Algorithm_HYSPLIT.py chart series HailCreek 09956 010302 GFSQ 2019091404 2019091500 2019091504

CSF

“Algorithm_CSY.py” package contains implementation of CSF algorithm. This algorithm uses the TROPOMI file and corresponding ERA5 files which need to be downloaded locally.

You can run CSF algorithm using the following command for orbit 09956:

python .\Algorithm_CSF.py run HailCreek 09956 010302

On completion you can list outputs of CSF using the following command:

python .\Algorithm_CSF.py list HailCreek 09956 010302

This will generate an output that looks like:

Orbit: 09956 Processor: 010302 Date: 2019-09-15 03:49
Wind: u: -5.6952834868095294 (m/s) v: -1.0697813437018595 (m/s) speed: 5.794884478439399 (m/s)
Configuration
        Background geometry: Geometry.CONTAINS
        Background minimum pixel count: 20
        Downwind box geometry: Geometry.CONTAINS
        Downwind box mask: Mask.NONE
        Downwind box enhancement delta: 5
        Transect geometry: algorithm
Background
        Geometry: CONTAINS
        Upwind box: Count: 31 Upwind Average: 1812.3728578629032
        Domain median: 1806.881591796875
        Manual background: Not used
        Background enhancement: 1812.3728578629032
        Background downwind box: 1812.3728578629032
Downwind box
        Geometry: CONTAINS Mask: NONE
        Rotation: 5.0
        Length: 99000.0 (m) Width: 44000.0 (m)
Transects
        Valid pixels count: 104 Valid positive pixel count: 62
        Valid transect count: 11 Minimum: 1795.307373046875 (ppb)
        Valid: True Enh:  0.560(kg/m) Length: [masked: 0.000(m) valid: 43921.470(m) positive: 22097.327(m)] Count: [masked: 0 valid: 9 positive: 5] Line: [148.180,-21.707] - [148.139,-21.312] 
        Valid: True Enh:  1.138(kg/m) Length: [masked: 0.000(m) valid: 43910.675(m) positive: 34066.129(m)] Count: [masked: 0 valid: 10 positive: 8] Line: [148.112,-21.714] - [148.071,-21.318]
        Valid: True Enh:  1.096(kg/m) Length: [masked: 0.000(m) valid: 43910.055(m) positive: 22418.543(m)] Count: [masked: 0 valid: 10 positive: 5] Line: [148.044,-21.720] - [148.003,-21.324]
        Valid: True Enh:  0.514(kg/m) Length: [masked: 0.000(m) valid: 43909.287(m) positive: 19750.431(m)] Count: [masked: 0 valid: 10 positive: 5] Line: [147.976,-21.726] - [147.935,-21.330]
        Valid: True Enh:  1.819(kg/m) Length: [masked: 0.000(m) valid: 43910.104(m) positive: 37276.487(m)] Count: [masked: 0 valid: 10 positive: 8] Line: [147.909,-21.732] - [147.867,-21.336]
        Valid: True Enh:  1.435(kg/m) Length: [masked: 0.000(m) valid: 43909.117(m) positive: 21553.540(m)] Count: [masked: 0 valid: 10 positive: 5] Line: [147.841,-21.738] - [147.799,-21.343]
        Valid: True Enh:  2.363(kg/m) Length: [masked: 0.000(m) valid: 43920.729(m) positive: 35744.508(m)] Count: [masked: 0 valid: 9 positive: 7] Line: [147.773,-21.744] - [147.731,-21.349]
        Valid: True Enh:  1.216(kg/m) Length: [masked: 0.000(m) valid: 43910.189(m) positive: 34069.936(m)] Count: [masked: 0 valid: 10 positive: 7] Line: [147.705,-21.750] - [147.663,-21.355]
        Valid: True Enh:  4.544(kg/m) Length: [masked: 0.000(m) valid: 43910.525(m) positive: 43910.525(m)] Count: [masked: 0 valid: 9 positive: 9] Line: [147.637,-21.756] - [147.595,-21.361]
        Valid: True Enh:  2.089(kg/m) Length: [masked: 0.000(m) valid: 43909.842(m) positive: 33799.922(m)] Count: [masked: 0 valid: 9 positive: 6] Line: [147.569,-21.762] - [147.527,-21.367]
        Valid: True Enh:  0.298(kg/m) Length: [masked: 0.000(m) valid: 43909.610(m) positive: 16900.092(m)] Count: [masked: 0 valid: 9 positive: 3] Line: [147.501,-21.768] - [147.459,-21.372]
        Valid: False Enh:  0.000(kg/m) Length: [masked: 0.000(m) valid: 43920.880(m) positive: 0.000(m)] Count: [masked: 0 valid: 8 positive: 0] Line: [147.433,-21.774] - [147.391,-21.378]
Emission rate: 32.25627636909812 (t/hour) 282.5649809932995 (Gg/year)
Status: SUCCESS

The following commands provide chart utilities of CSF. Each of these will chart a different aspect of the algorithm:

python .\Algorithm_CSF.py chart valid HailCreek 09956 010302

python .\Algorithm_CSF.py chart positive HailCreek 09956 010302

python .\Algorithm_CSF.py chart transects HailCreek 09956 010302

IME

“Algorithm_IME.py” package contains implementation of IME algorithm. This algorithm uses the output of the CSF. Outputs of IME can be listed, for the 12th transect, using the following command:

python .\Algorithm_IME.py list HailCreek 09956 010302 12

This will generate an output that looks like:

Transect ID:12, Distance to source: 77.783(km)
Wind:20.782 (km/h), Residence time: 3.743(h)
Background:1812.3728578629032(ppb), IME:100242.126(kg), Q: 26.783(t/hour) = 234.619(Gg/year), IEF:27.394

Similarly, outputs of IME can be charted using the following command:

python .\Algorithm_IME.py chart HailCreek 09956 010302 12

TM

“Algorithm_TM.py” package contains implementation of TM algorithm. This algorithm uses the output of the CSF and HYSPLIT algorithms.

Outputs of TM can be listed for background 1812.373(ppb) using the following command:

python .\Algorithm_TM.py list HailCreek 09956 GFSQ 010302 1812.373

This will generate output that looks like:

background:1812.373(ppb), tm:131504.843(kg), Q:7.306(t/hour) = 63.999(Gg/year), IEF:7.473

Similarly, a chart can be generated using the following command:

python .Algorithm_TM.py chart HailCreek 09956 010302 GFSQ 1812

Paper

“Paper.py” contains functions used to generate data used in this report. It relies on existence of all data used, including TROPOMI files, charts from BoM, Moranbah AWS data, ERA5 data files, HYSPLIT modelling outputs and SRTM files.

To list all orbits included use the following command:

python .\Paper.py list csf summary figure2_all

To list all orbits included use the following command:

python .\Paper.py list csf summary figure2_min_max