Estimate the Area of an Image Segment

· 5 min read

Using Basic ML with Wolfram Engine

Hi there! I am working as a spectroscopist. We shine light through the samples and estimate how much is absorbed and at which frequency. In this case, it is quite crucial to know the light spot, which is collimated by a special copper aperture:

Download original notebook

Let's drag and drop our image and adjust the brightness:

Import["yourPathToDroppedImage.jpeg"];
ImageAdjust[ImageResize[%, 400], {0.01, 2}]

Here is my sample image:

img = (*VB[*)(FrontEndRef["29e2ab9f-8a3d-492a-a77a-13df130878b9"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKG1mmGiUmWabpWiQap+iaWBol6iaamyfqGhqnpBkaG1iYWyRZAgCJ+xWr"*)(*]VB*);

Image segmentation

There is a local neural network, which is ready to be used for a general image segmentation:

(segments = ImageSegmentationComponents[img]) // AbsoluteTiming;
Quantity[% // First, "Seconds"]
segments // Colorize 

(*VB[*)(Quantity[63.408851, "Seconds"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KWnMIB4vkAjLTC13SU3OL0osyS8K5gCKBJYm5pVkllQWvfl+xCp0i79DMDtQNBioJi+lGADByRWf"*)(*]VB*)

(*VB[*)(FrontEndRef["33a9e052-8529-40e9-bb21-d580107ea254"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKGxsnWqYamBrpWpgaWeqaGKRa6iYlGRnqpphaGBgamKcmGpmaAAB1TxTa"*)(*]VB*)

It takes a lot of time! Can we do better for our case?

Morphological components

MorphologicalComponents[img] // Colorize

(*VB[*)(FrontEndRef["8f24228f-919c-4006-9f65-a8eca3335f86"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKW6QZmRgZWaTpWhpaJuuaGBiY6VqmmZnqJlqkJicaGxubplmYAQB4JhUn"*)(*]VB*)

Wow cool. It is faster than CNN (I guess), but requires some tweaking:

MorphologicalComponents[img, 0.5] // Colorize

(*VB[*)(FrontEndRef["e584841b-1ba8-4ff7-ba7f-0a8f79268bab"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKp5pamFiYGCbpGiYlWuiapKWZ6yYlmqfpGiRapJlbGplZJCUmAQCHXRYP"*)(*]VB*)

Nope 🥺 I did not manage to make it work with a simpler approach. I guess, we have to stick to CNN.

Back to CNN

Let's pick a point on our image:

HighlightImage[img, {Red, Point[(*BB[*)({268,177.65625`})(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRAeB5AILqnMSXXKr0hjgskHleakFnMBGU6JydnpRfmleSlpzDDlQe5Ozvk5+UVFDGDwwR6dwcAAAAHdFiw="*)(*]BB*)]}]

(*VB[*)(FrontEndRef["2d6fd627-6c9f-469a-8add-1613a1062c9a"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKG6WYpaWYGZnrmiVbpumamFkm6lokpqToGpoZGicaGpgZJVsmAgCHARWn"*)(*]VB*)

tip

Do not do by repeated revaluation of the same cell. Use Command Palette tool ==Inline navigation drag gizmo== while selecting the coordinates of a point highlighted in yellow

Now let us try again:

segments = ImageSegmentationComponents[img];

Select one with a minimal area and one that includes our point:

hole = With[{i = SortBy[Select[ComponentMeasurements[{img, segments}, "BoundingBox"], RegionMember[Rectangle @@ #[[2]], (*BB[*)({268,177.65625`})(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRAeB5AILqnMSXXKr0hjgskHleakFnMBGU6JydnpRfmleSlpzDDlQe5Ozvk5+UVFDGDwwR6dwcAAAAHdFiw="*)(*]BB*)]&], Area[Rectangle @@ #]&]//Last//First},
  SelectComponents[{img, segments}, (#Label == i) &] // Last 
];

hole // Colorize 

(*VB[*)(FrontEndRef["c5cbc3ab-ac8a-485d-993d-54d53cccc5e5"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKJ5smJyUbJybpJiZbJOqaWJim6FpaGqfompqkmBonA4FpqikAmiAWlA=="*)(*]VB*)

Measure the area

ComponentMeasurements[hole, "Area"] // First // Last

1358.875`

Calibration

To get the correct value in real units, we need to adjust the integrated value by the corresponding Jacobian. Assuming that perspective distortions are negligible and the aspect ratio of the image was not changed, the area can be corrected by a constant multiplier corresponding to:

\text{Area} = \text{UnitsPerPixel}^2 \cdot \text{PixelArea}

We simply place two points using gizmo helpers:

HighlightImage[img, {Red, Line[{{129,24.828125000000018`}, {129,262.828125`}}]}]

(*VB[*)(FrontEndRef["539e7a14-15ed-475e-8766-e19f8116320b"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKmxpbpponGproGpqmpuiamJum6lqYm5npphpaplkYGpoZGxkkAQB5tRTx"*)(*]VB*)

Esimate units per pixel

Line[{{129,24.828125000000018`}, {129,262.828125`}}] // ArcLength;
20.0 / %

0.08403361344537816`

Final result: area of the hole

HighlightImage[Blend[{img, Colorize[hole]}, 0.5], 
  {Directive[Red, FontSize->18], Text[StringTemplate["Area: `` mm^{2}"][(*SpB[*)Power[0.08403(*|*),(*|*)2](*]SpB*) 1358.87], {268,177.65625 + 35}, {0,0}]}
]

(*VB[*)(FrontEndRef["83b999da-39b4-4e04-9c78-32b37c57299a"])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KRCeEJBwK8rPK3HNS3GtSE0uLUlMykkNVgEKWxgnWVpapiTqGlsmmeiapBqY6Fomm1voGhslGZsnm5obWVomAgB+BxUv"*)(*]VB*)

Automate it!

Let's create a widget that can accomplish this task in a few steps:

Overlay

Here, we overlay our image with an inset that includes a dynamic Image. We reload the image data with masked regions depending on where the user's cursor is pointing:

Module[{
  overlay, 
  segment = 1
}, With[{
  masks = Values[ComponentMeasurements[{img, segments}, {"Mask", "BoundingBox", "Label", "Area", "BoundingBoxArea"}]]
},
  
  overlay = ImageData[Colorize[masks[[segment, 1]]], "Byte"];

  EventHandler[HighlightImage[Blend[{img, Colorize[segments]}, 0.4], {
    {Opacity[0.2], Inset[Image[overlay // Offload, "Byte"], Center, Center]}
  }], {"mousemove" -> Function[xy, 
    With[{
      r = SortBy[Select[masks, RegionMember[Rectangle @@ #[[2]], xy]&], Last] // First // Quiet
    },
      If[!ListQ[r], Return[]];
      If[r[[3]] == segment, Return[]];
      
      segment = r[[3]];
      overlay = ImageData[Colorize[masks[[segment, 1]]], "Byte"];
    ]
  ]}]

]]

note

Here we pick a segment with a minimal bounding box area to discard tiny distributed non-connected features.

Virtual Ruler

We need to match the scale of the image with real units. For that resound we construct two handles to place two points on the image. The easiest one would be this:

Graphics[{
  Line[{{0,1}, {0,-1}}], Line[{{-0.3,1}, {0.3,1}}], Line[{{-0.3,-1}, {0.3,-1}}],
  Opacity[0.5], Red, Rectangle[{0,1}-{0.2,0.2}, {0,1}+{0.2,0.2}], Rectangle[{0,-1}-{0.2,0.2}, {0,-1}+{0.2,0.2}]
}, PlotRange->1.2{{-1,1}, {-1,1}}, ImageSize->Small]

(*VB[*)(Graphics[{Line[{{0, 1}, {0, -1}}], Line[{{-0.3, 1}, {0.3, 1}}], Line[{{-0.3, -1}, {0.3, -1}}], Opacity[0.5], RGBColor[1, 0, 0], Rectangle[{-0.2, 0.8}, {0.2, 1.2}], Rectangle[{-0.2, -1.2}, {0.2, -0.8}]}, PlotRange -> {{-1.2, 1.2}, {-1.2, 1.2}}, ImageSize -> Small])(*,*)(*"1:eJxTTMoPSmNkYGAoZgESHvk5KWnMIB4HkHAvSizIyEwuTmOHyftkFpcgVPtk5qWmMaHIofAygTRDJkg5NvH/QEC8WUXGYHB5PxbzYHL2EDkyzAS7BYeZCHeCgsG/IDE5s6SyiAEMHtgjQivI3ck5Pye/COwIiBcZ4E7lBClITS5JzEvPQXNT0ayZIHByP5Tx0h67vD3USZ/tSTASqmU/LiOhViLJB5XmpILNDsjJLwkCGk5MEH7ej+Y4YuThNnnmJqanBmdWpRazAnnBuYk5OQCX9Kl6"*)(*]VB*)

Where two red rectangles can be used as dragging area for a user mouse. Of course we have to recalculate the posiitons of upper and bottom segments considering the angle, i.e. not only for horisontal or vertical directions.

Module[{
  overlay, 
  segment = 1, 
  nav = {{0.5,0.6} ImageDimensions[img], {0.5,0.4} ImageDimensions[img]},
  minScale = {0.1,0.1} ImageDimensions[img] / 4.0
}, With[{
  masks = Values[ComponentMeasurements[{img, segments}, {"Mask", "BoundingBox", "Label", "Area", "BoundingBoxArea"}]],
  helperLines = Function[p, With[{dir = Normalize[p[[2]]-p[[1]]]},
    {p, 
      {p[[1]] - minScale[[1]]  {dir[[2]], -dir[[1]]}, p[[1]] + minScale[[1]]  {dir[[2]], -dir[[1]]}}, 
      {p[[2]] - minScale[[1]]  {dir[[2]], -dir[[1]]}, p[[2]] + minScale[[1]]  {dir[[2]], -dir[[1]]}}
    }
  ]]
},

  nav = helperLines[nav];
  
  overlay = ImageData[Colorize[masks[[segment, 1]]], "Byte"];

  EventHandler[HighlightImage[Blend[{img, Colorize[segments]}, 0.4], {

    {Opacity[0.2], Inset[Image[overlay // Offload, "Byte"], Center, Center]},
  
    Cyan, Line[nav[[1]] // Offload], 
    Line[nav[[2]] // Offload], Line[nav[[3]] // Offload],

    { Opacity[0],
      EventHandler[Rectangle[nav[[1,1]] - minScale, nav[[1,1]] + minScale], {"drag"->Function[xy,
        nav[[1, 1]] = xy;
        nav = helperLines[nav[[1]]];
      ]}],
      EventHandler[Rectangle[nav[[1,2]] - minScale, nav[[1,2]] + minScale], {"drag"->Function[xy,
        nav[[1, 2]] = xy;
        nav = helperLines[nav[[1]]];
      ]}]
    }
    
  }], {"mousemove" -> Function[xy, 
    With[{
      r = SortBy[Select[masks, RegionMember[Rectangle @@ #[[2]], xy]&], Last] // First // Quiet
    },
      If[!ListQ[r], Return[]];
      If[r[[3]] == segment, Return[]];
      
      segment = r[[3]];
      overlay = ImageData[Colorize[masks[[segment, 1]]], "Byte"];
    ]
  ]}]

]]

Final version

Let's add a panel with a slider to enter the length of the measured distance in real-world units and a text window with calculated area:

Module[{
  overlay, 
  segment = 1, 
  distance = 1.0,
  area = 0.0,
  nav = {{0.5,0.6} ImageDimensions[img], {0.5,0.4} ImageDimensions[img]},
  minScale = {0.1,0.1} ImageDimensions[img] / 4.0
}, With[{
  masks = Values[ComponentMeasurements[{img, segments}, {"Mask", "BoundingBox", "Label", "Area", "BoundingBoxArea"}]],
  helperLines = Function[p, With[{dir = Normalize[p[[2]]-p[[1]]]},
    {p, 
      {p[[1]] - minScale[[1]]  {dir[[2]], -dir[[1]]}, p[[1]] + minScale[[1]]  {dir[[2]], -dir[[1]]}}, 
      {p[[2]] - minScale[[1]]  {dir[[2]], -dir[[1]]}, p[[2]] + minScale[[1]]  {dir[[2]], -dir[[1]]}}
    }
  ]],

  computeArea = Function[{d,p,m,s}, (*SpB[*)Power[((*FB[*)((d)(*,*)/(*,*)(Norm[p[[1,1]]-p[[1,2]]]))(*]FB*))(*|*),(*|*)2](*]SpB*) m[[s, 4]]]
},

  nav = helperLines[nav];
  area = computeArea[distance, nav, masks, segment];
  
  overlay = ImageData[Colorize[masks[[segment, 1]]], "Byte"];

 {
   EventHandler[HighlightImage[Blend[{img, Colorize[segments]}, 0.4], {

    {Opacity[0.2], Inset[Image[overlay // Offload, "Byte"], Center, Center]},
  
    Cyan, Line[nav[[1]] // Offload], 
    Line[nav[[2]] // Offload], Line[nav[[3]] // Offload],

    { Opacity[0],
      EventHandler[Rectangle[nav[[1,1]] - minScale, nav[[1,1]] + minScale], {"drag"->Function[xy,
        nav[[1, 1]] = xy;
        nav = helperLines[nav[[1]]];
      ]}],
      EventHandler[Rectangle[nav[[1,2]] - minScale, nav[[1,2]] + minScale], {"drag"->Function[xy,
        nav[[1, 2]] = xy;
        nav = helperLines[nav[[1]]];
      ]}]
    }
    
  }], {"mousemove" -> Function[xy, 
    With[{
      r = SortBy[Select[masks, RegionMember[Rectangle @@ #[[2]], xy]&], Last] // First // Quiet
    },
      If[!ListQ[r], Return[]];
      If[r[[3]] == segment, Return[]];
      
      segment = r[[3]];
      overlay = ImageData[Colorize[masks[[segment, 1]]], "Byte"];
      area = computeArea[distance, nav, masks, segment];
    ]
  ]}],
  
  Panel[{
    EventHandler[InputRange[0, 50, 0.01, 1.0, "Label"->"Distance"], Function[d, 
      distance = d;
      area = computeArea[distance, nav, masks, segment];
    ]],

    TextView[area // Offload, "Label"->"Area"]
  } // Column]
 } // Row 

]]

Final final version

As a last commit we can add a file browser and contrast/brightness correction widget:

widget1[img_Image] := Module[{
  overlay, 
  segment = 1, 
  distance = 1.0,
  area = 0.0,
  nav = {{0.5,0.6} ImageDimensions[img], {0.5,0.4} ImageDimensions[img]},
  minScale = {0.1,0.1} ImageDimensions[img] / 4.0,
  segments = ImageSegmentationComponents[img]
}, With[{
  masks = Values[ComponentMeasurements[{img, segments}, {"Mask", "BoundingBox", "Label", "Area", "BoundingBoxArea"}]],
  helperLines = Function[p, With[{dir = Normalize[p[[2]]-p[[1]]]},
    {p, 
      {p[[1]] - minScale[[1]]  {dir[[2]], -dir[[1]]}, p[[1]] + minScale[[1]]  {dir[[2]], -dir[[1]]}}, 
      {p[[2]] - minScale[[1]]  {dir[[2]], -dir[[1]]}, p[[2]] + minScale[[1]]  {dir[[2]], -dir[[1]]}}
    }
  ]],

  computeArea = Function[{d,p,m}, (*SpB[*)Power[((*FB[*)((d)(*,*)/(*,*)(Norm[p[[1,1]]-p[[1,2]]]))(*]FB*))(*|*),(*|*)2](*]SpB*) m[[4]]]
},

  nav = helperLines[nav];
  area = computeArea[distance, nav, masks[[segment]]];
  
  overlay = ImageData[Colorize[masks[[segment, 1]]], "Byte"];

 {
   EventHandler[HighlightImage[Blend[{img, Colorize[segments]}, 0.4], {

    {Opacity[0.2], Inset[Image[overlay // Offload, "Byte"], Center, Center]},
  
    Cyan, Line[nav[[1]] // Offload], 
    Line[nav[[2]] // Offload], Line[nav[[3]] // Offload],

    { Opacity[0],
      EventHandler[Rectangle[nav[[1,1]] - minScale, nav[[1,1]] + minScale], {"drag"->Function[xy,
        nav[[1, 1]] = xy;
        nav = helperLines[nav[[1]]];
      ]}],
      EventHandler[Rectangle[nav[[1,2]] - minScale, nav[[1,2]] + minScale], {"drag"->Function[xy,
        nav[[1, 2]] = xy;
        nav = helperLines[nav[[1]]];
      ]}]
    }
    
  }], {"mousemove" -> Function[xy, 
    With[{
      r = SortBy[Select[masks, RegionMember[Rectangle @@ #[[2]], xy]&], Last] // First // Quiet
    },
      If[!ListQ[r], Return[]];
      If[r[[3]] == segment, Return[]];
      
      segment = r[[3]];
      overlay = ImageData[Colorize[masks[[segment]][[1]]], "Byte"];
      area = computeArea[distance, nav, masks[[segment]]];
    ]
  ]}],
  
  Panel[{
    EventHandler[InputRange[0, 50, 0.01, 1.0, "Label"->"Distance"], Function[d, 
      distance = d;
      area = computeArea[distance, nav, masks[[segment]]];
    ]],

    TextView[area // Offload, "Label"->"Area"]
  } // Column]
 } // Row 

]];

widget2[img_] := Module[{
  imageData = ImageData[img, "Byte"]
}, With[{o = img},

  {
    Image[imageData // Offload, "Byte"],
    EventHandler[InputGroup[{
      InputRange[-1,2,0.1, "Label"->"Contrast"],
      InputRange[0,2,0.1, "Label"->"Brightness"]
    }], Function[c,
      img = ImageAdjust[o, c];
      imageData = ImageData[img, "Byte"];
    ]]
  } // Row

]];

SetAttributes[widget2, HoldFirst]

Evaluate this cell

path = SystemDialogInput["FileOpen"];
img = Null;

If[ImageQ[img = Import[path]],
  img = ImageResize[img, 350];
  widget2[img]
, $Failed]

After adjusting evaluate this one

widget1[img]

Convert to a Mini App

You can export this entire notebook into a mini-app. There are a few rules to follow:

All initialization cells will be executed on start.
The last input cell will represent the main window of the app.
Any defined variables in the notebook will be isolated.

Since we have two widgets, it might be handy to work with multiple windows.

Main Window

This will just be a file browser button. But first, define some helper components:

.wlx

progressBarIcon = <svg aria-hidden="true" class="w-5 h-4 text-gray-200 animate-spin dark:text-gray-600 fill-teal-600" viewBox="0 0 100 101" fill="none"><path d="M100 50.5908C100 78.2051 77.6142 100.591 50 100.591C22.3858 100.591 0 78.2051 0 50.5908C0 22.9766 22.3858 0.59082 50 0.59082C77.6142 0.59082 100 22.9766 100 50.5908ZM9.08144 50.5908C9.08144 73.1895 27.4013 91.5094 50 91.5094C72.5987 91.5094 90.9186 73.1895 90.9186 50.5908C90.9186 27.9921 72.5987 9.67226 50 9.67226C27.4013 9.67226 9.08144 27.9921 9.08144 50.5908Z" fill="currentColor"></path><path d="M93.9676 39.0409C96.393 38.4038 97.8624 35.9116 97.0079 33.5539C95.2932 28.8227 92.871 24.3692 89.8167 20.348C85.8452 15.1192 80.8826 10.7238 75.2124 7.41289C69.5422 4.10194 63.2754 1.94025 56.7698 1.05124C51.7666 0.367541 46.6976 0.446843 41.7345 1.27873C39.2613 1.69328 37.813 4.19778 38.4501 6.62326C39.0873 9.04874 41.5694 10.4717 44.0505 10.1071C47.8511 9.54855 51.7191 9.52689 55.5402 10.0491C60.8642 10.7766 65.9928 12.5457 70.6331 15.2552C75.2735 17.9648 79.3347 21.5619 82.5849 25.841C84.9175 28.9121 86.7997 32.2913 88.1811 35.8758C89.083 38.2158 91.5421 39.6781 93.9676 39.0409Z" fill="currentFill"></path></svg>

<svg aria-hidden="true" class="w-5 h-4 text-gray-200 animate-spin dark:text-gray-600 fill-teal-600" viewBox="0 0 100 101" fill="none"><path d="M100 50.5908C100 78.2051 77.6142 100.591 50 100.591C22.3858 100.591 0 78.2051 0 50.5908C0 22.9766 22.3858 0.59082 50 0.59082C77.6142 0.59082 100 22.9766 100 50.5908ZM9.08144 50.5908C9.08144 73.1895 27.4013 91.5094 50 91.5094C72.5987 91.5094 90.9186 73.1895 90.9186 50.5908C90.9186 27.9921 72.5987 9.67226 50 9.67226C27.4013 9.67226 9.08144 27.9921 9.08144 50.5908Z" fill="currentColor"></path><path d="M93.9676 39.0409C96.393 38.4038 97.8624 35.9116 97.0079 33.5539C95.2932 28.8227 92.871 24.3692 89.8167 20.348C85.8452 15.1192 80.8826 10.7238 75.2124 7.41289C69.5422 4.10194 63.2754 1.94025 56.7698 1.05124C51.7666 0.367541 46.6976 0.446843 41.7345 1.27873C39.2613 1.69328 37.813 4.19778 38.4501 6.62326C39.0873 9.04874 41.5694 10.4717 44.0505 10.1071C47.8511 9.54855 51.7191 9.52689 55.5402 10.0491C60.8642 10.7766 65.9928 12.5457 70.6331 15.2552C75.2735 17.9648 79.3347 21.5619 82.5849 25.841C84.9175 28.9121 86.7997 32.2913 88.1811 35.8758C89.083 38.2158 91.5421 39.6781 93.9676 39.0409Z" fill="currentFill"></path></svg>

Now our window and it's logic:

notebook = EvaluationNotebook[];
img = Null;

ClearAll[MainWindow];

MainWindow := Panel[{
  Style["Open an image", 10, Lighter[Black],FontFamily->"system-ui"],
  EventHandler[
    InputButton["Browse"], 
    AsyncFunction[Null, Module[{path, cell, progressBar = ""}, 
      path = SystemDialogInputAsync["FileOpen"] // Await;
      If[StringQ[path],
        With[{i = Import[path]},
          If[ImageQ[i], 
            img = ImageResize[i, 350];
            cell = CellPrint[
             Panel[{
              widget2[img], {
              EventHandler[InputButton["Continue"], Function[Null,
                  If[progressBar =!= "", Return[]];
                  progressBar = progressBarIcon;
                  CellPrint[
                    Panel[widget1[img]], 
                    "Target"->_, "Notebook"->notebook, 
                    ImageSize->{800,500}
                  ];
                  Delete[cell];
                  ClearAll[progressBar];
                ]
              ], 
              HTMLView[progressBar // Offload]
            } //Row 
           } //Column ]
        , "Target"->_, "Notebook"->notebook, ImageSize->{800,500}]
      ]
    ]
  ];
  
  ClearAll[path];
]]]} // Column]

As a final step - export this notebook to a mini app using Share menu

MainWindow

Image segmentation​

Morphological components​

Back to CNN​

Calibration​

Automate it!​

Overlay​

Virtual Ruler​

Final version​

Final final version​

Convert to a Mini App​

Main Window​