TDS Interactive Signal Processing

⏱️ 6 min read

Maintainer

It is common in time-domain spectroscopy to observe copies of the initial system response due to impedance mismatch. If deconvolution is not possible using slab-like models, then we have to smoothly suppress them to avoid artifacts in further processing.

We perform a 'surgery' by applying an envelope function to the signal to suppress echoes, which cause oscillations in the frequency domain:

Download original notebook

Key Considerations:

Cutting reduces resolution, so we use exponential decay to avoid artifacts
We wrap everything with Manipulate to interactively tune the parameters

Test spectrum processing will follow:

test = NotebookStore["troubling-fd8"];

{
  ListLinePlot[test, PlotRange->Full, PlotLabel->"Time domain"], 
  ListLinePlot[Take[Fourier[test[[All,2]]], 500] // Abs, PlotLabel->"Frequency domain" ]
} // Row 

(*GB[*){{(*VB[*)(FrontEndRef["6dd01e13-26a1-4ae2-a59d-7c98d9443769"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKm6WkGBimGhrrGpklGuqaJKYa6SaaWqbomidbWqRYmpgYm5tZAgCCuhVN"*)(*]VB*)(*|*),(*|*)(*VB[*)(FrontEndRef["e6ccd935-7527-4397-a0f5-4a94cd7cacd2"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKp5olJ6dYGpvqmpsameuaGFua6yYapJnqmiRamiSnmCcnJqcYAQCDpBXe"*)(*]VB*)}}(*]GB*)

One can see the wiggles in the fourier image, which we should get rid of without any loss of resolution.

Let's define our envelope generator:

env[offset_, \[Sigma]_] := Function[x, If[x>=offset, Exp[- (*FB[*)(((x-offset))(*,*)/(*,*)(\[Sigma]))(*]FB*)], 1.0]]

tip

Instead of direct calculation of the envelope we return a pure function to speed up the calculations later. Pattern matching is fast, but pure functions wrapped with Map or Table can take advantage of just-in-time compiler.

Now we apply to it to our time trace and perform Fourier transformation interactively

Manipulate[With[
{
  e = env[offset, \[Sigma]]
},
{
  filtered = Map[Function[xy,
      {xy[[1]], xy[[2]]  e @ xy[[1]]}
  ], test],
  envelope = Map[Function[xy,
      {xy[[1]], 10.0 e @ xy[[1]]}
  ], test]
},

Row[{
  ListLinePlot[{
    (*BB[*)(* skip every 2nd point *)(*,*)(*"1:eJxTTMoPSmNhYGAo5gcSAUX5ZZkpqSn+BSWZ+XnFaYwgCS4g4Zyfm5uaV+KUXxEMUqxsbm6exgSSBPGCSnNSg9mAjOCSosy8dLBYSFFpKpoKkDkeqYkpEFXBILO1sCgJSczMQVYCAOFrJEU="*)(*]BB*)
    test[[;;;;2]],
    filtered[[;;;;2]],
    envelope[[;;;;2]]
  }, PlotRange->Full]
,
  ListLinePlot[{
    Take[Fourier[test[[All,2]]]//Abs, 500],
    Take[Fourier[filtered[[All,2]]]//Abs, 500]
  }, PlotRange->Full]
}]

], {{offset, 1108}, 1100, 1114, 2}, {{\[Sigma], 0.4}, 0.2, 0.6, 0.1}, ContinuousAction->True, Appearance->None]

(*VB[*)(FrontEndRef["2a210068-a67b-4c27-bdfc-63d1b48ea415"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKGyUaGRoYmFnoJpqZJ+maJBuZ6yalpCXrmhmnGCaZWKQmmhiaAgB+PhWS"*)(*]VB*)

Here you can see, that by tuning the offset parameter and damping $1/\sigma$ we can nicely remove all the wiggles.

Batch processing

If you have many similar files, you can import them one by one and process on fly using the same envelope function

With[{
  original = Import[#, HeaderLines->1],
  e = env[1108, 0.4]
},
{
  new = Map[Function[xy,
      {xy[[1]], xy[[2]] e @ xy[[1]]}
  ], original]
},  
  Export[FileNameJoin[{
    DirectoryName[#], "Filtered", FileNameTake[#]
  }], new]
] &/@ FileNames["*.csv", "path_to_your_files"];
Pane[Style[% // TableForm, 10, Italic], {800,300}]

(*BB[*)((*BB[*)((*GB[*){{"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_100K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_10K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_110K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_120K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_130K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_140K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_150K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_160K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_170K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_180K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_190K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_200K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_20K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_210K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_230K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_250K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_270K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_280K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_290K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_300K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_300K_ref.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_30K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_40K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_50K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_5K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_5K_ref.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_60K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_6K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_70K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_80K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIa_90K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_100K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_10K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_110K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_120K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_130K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_140K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_150K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_160K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_170K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_180K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_190K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_200K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_20K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_210K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_230K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_250K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_270K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_280K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_290K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_300K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_300K_ref.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_30K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_40K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_50K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_5K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_5K_ref.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_60K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_6K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_70K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_80K.csv"}(*||*),(*||*){"C:\\Messdaten\\Daten Teraview Modernazed\\SmFeO\\ac\\Filtered\\EIIc_90K.csv"}}(*]GB*))(*,*)(*"1:eJxTTMoPSmNmYGAo5gMSwSWVOakuqcn5RYkl+UWZXCBxNiDhWZKYk5mcxgTiswCJoNKc1GIesIaizAL/PM+8gtKSYlaggFtiTnEqAHHDFV0="*)(*]BB*))(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KWlMIB4vkAjLTC13SU3OL0osyS8KBskHJOalQuRBvKDSnNRiTiDDMzcxPTU4swpJziezuCRTgZmBIVMHaDoAeyIVuQ=="*)(*]BB*)

Batch processing​

Batch processing