Planteome Image Annotation Working Group

Goals and Objectives

---

Detailed Software Specification

Tentative Development Timeline (based on current staffing and resources)

More about AISO, plus downloads.

Phase 1: Spring-Summer 2015

Objective: Implement asynchronous user-directed graph cuts in the Bisque environment.

Summary

• Build a Bisque module that allows the user to “mark up” an uploaded image, submit that markup to the Bisque server for graph cut segmentation, and receive the segmented results as a layered annotation.
• Build and integrate a separate feature into the metadata space to allow annotation of image segments with ontology terms (eventually provided remotely via web service.)

Phase 2: Summer-Fall-Winter 2015

Objective: Implement “wired” ontology metadata. Interactive, single-session graph cut segmentation of multiple, potentially overlapping segments on one image.

Summary:

• Allow the user to provide multiple mark-up lines and perform one or more segmentations on an image in near-real time, during a single user session. Will also allow revision tracking for multiple undo-redo steps as required by the user.
• Enhance the segmentation feature to render and manage multiple segment objects on a single image, with appropriate overlap, translucency, and inter-segment navigation capabilities.

Phase 3: 2016-2017

Objective(s):

• Implement machine-learning (“active learning”) algorithms to optimize segmentation and to auto-annotate plant images
• Optimize for very large, hi-res images (tiling? scaling?)
• Enable use of localized ontology reference data (a la custom ontologies provided by the user, or in an uploaded file)

Participants

• Jaiswal Lab (OSU, BPP): Justin Preece, Justin Elser
• Todorovic Group (OSU, EECS): Sinisa Todorovic, Xu Xu, Ryan Kitchen (Yao Zhou has left the project)
• iPlant-Bisque Group (UCSB): Kris Kvilekval, Dmitri Federov, B.S. Manjunath

Image Annotation Working Group Meetings

Initial Project Meeting: Jan 23, 2015

(Conf. call arranged and hosted on Zoom by Martha Narro, iPlant)

Attendees

• Jaiswal Lab (OSU, BPP): Pankaj Jaiswal, Justin Preece
• Todorovic Group (OSU, EECS): Sinisa Todorovic, Yao Zhou
• iPlant-Bisque Group (UCSB): Kris Kvilekval, Dmitri Federov, B.S. Manjunath

Agenda

• (Re-)Introductions
• Planteome AISO/Bisque proposal review
• Discussion of broad project objectives
• Scheduled next meeting (will be a technical meeting)

Action Items

• Justin send Yao the AISO documentation
• Kris and Dmitry send Yao (and all) pointers to Bisque documentation and Google group.
• Yao: begin to get familiar with Bisque and the AISO Java codebase.

AISO Demo (OSU Only): Tue, Jan 27, 1 PT

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece
• Todorovic Group (OSU, EECS): Sinisa Todorovic, Yao Zhou, Xu Xu

Agenda

• Demos and technical introductions to AISO and codebase
• Preliminary list of features desired to port to Bisque

Tech Demo Meeting: Thu, Jan 29, noon PT/1p MT

(Conf. call arranged and hosted on Zoom by Martha Narro, iPlant)

Attendees

• Jaiswal Lab (OSU, BPP): Pankaj Jaiswal, Justin Preece
• Todorovic Group (OSU, EECS): Sinisa Todorovic, Yao Zhou, Xu Xu
• iPlant-Bisque Group (UCSB): Kris Kvilekval, Dmitri Federov

Agenda

• Demos and technical introductions to Bisque development environment and AISO functionality
  • Second grad student developer (Xu Xu) added from Sinisa's group
• Discuss IGC segmentation algorithm
• Discuss implementation of a Bisque ontology service
• Project planning and timelines
• Discussed possible attendance at May 14-15 Bisque developer workshop at U of Arizona

Tech Dev Meeting: Tue, Feb 10, 1p PT

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece
• Todorovic Group (OSU, EECS): Yao Zhou, Xu Xu

Agenda

• Discussed approaches to functionality ahead of Bisque tech mtg
  • Latency in Bisque's current round-trip approach to Graph Cuts segmentations - is it the round-trip or the graph cuts that takes so long? (JP thinks it's the round trip)
  • IGC segmentation algorithm - DCU and Bisque implementations different? (don't know yet)
  • Discuss implementation of segmentation module: general use-case
    • how to capture and transmit user input (foreground/background markup)
    • how to return segmentation results
    • emphasized importance of lean data communication, layered segments, overlap capabilities, undo/redo, how to store points sets and annotation data (in generic data structure terms)
• Project planning and timelines: define an iterative dev approach (multiple phases, with modest & clear initial goals)
  • will flesh out in mtg w/ Bisque team

Bisque Dev Env Setup Meeting: Thu, Feb 12, 1p MT

(Skype call)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece, Justin Elser
• Todorovic Group (OSU, EECS): Yao Zhou, Xu Xu
• iPlant-Bisque Group (UCSB): Kris Kvilekval, Dmitri Federov

Agenda

• Sort out issues in setting up Dev environment for YZ, XX
• Tour of Bisque module XML templates and sample MatLab image annotation script
• Define technical approach and requirements for the first phase of software development.
  • Discuss appropriate graph cuts algorithm (will probably use an implementation of Kolmogorov graph cuts
  • Discuss latency in Bisque's current round-trip approach to Graph Cuts segmentations

Action Items

• JP - draft requirements documentation for both the initial round of development, as well as more general objectives for the rest of the Planteome Image Annotation efforts
• All - review, discuss, and bicker over said requirements documentation.  :)
• YZ, XX, DF, KK - Troubleshoot dev environment setup for YZ and XX.
• Next mtg: TBD

PIA Phase 1 Specification Meeting: Fri, Mar 13, 2p PT

(Planteome Zoom call)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece
• Todorovic Group (OSU, EECS): Yao Zhou, Xu Xu
• iPlant-Bisque Group (UCSB): Kris Kvilekval

Agenda

• Revise specs for the first phase of software development
• Continue Dev environment setup with YZ, XX

Action Items

• JP - schedule next meeting for late next week (objectives: put timelines on Phase 1, discuss Phase 2/3 goals)
• KK - complete slimmed-down Bisque dev env deployment package for local installs
• All - think about rough timelines for Phase 1 activities
• YZ, XX, DF, KK - Cont. troubleshooting dev environment setup for YZ and XX.
• Next mtg: TBA (3/20?)

PIA Phase 1 Timeline Meeting: Thu, Apr 23, 10a PT

(Cord 3084)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece
• Todorovic Group (OSU, EECS): Sinisa Todorovic, Xu Xu, Ryan Kitchen

Agenda

• Discuss personnel changes: Yao Zhou is leaving the project. Ryan Kitchen (undergrad CS) is joining.
• Discuss timeline for Phase 1
• ST raised the issue of being able to run multiple visual analysis steps in a multi-step "pipeline" within this proposed module. Taken under advisement for future phases; and will factor into the initial architecture as much as possible.

Action Items

• JP - schedule official bi-weekly standing PIA developer meeting
• KK - complete slimmed-down Bisque dev env deployment package for local installs
• Next mtg: TBA

PIA Working Group Bi-Weekly Conf Call: Thu, Apr 30, 2p PT

(Zoom)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece
• Todorovic Group (OSU, EECS): Xu Xu, Ryan Kitchen
• UCSB / Bisque Group: Kris Kvilekval

Agenda

• Discuss installation of local Bisque engine package
• Discuss software development timeline and progress
• Prep for UCSB Workshop

Action Items

• XX and JP meet to prep prez and AISO demo for UCSB workshop

PIA Prez Prep Mtg: Tue, May 5, 11:30a PT

(Cord 3084)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece
• Todorovic Group (OSU, EECS): Xu Xu, Ryan Kitchen

Agenda

• Prep for UCSB Big Data + Bio-Image Workshop
• Fix AISO installation on MacBook

Action Items

• JP and XX to add and edit assigned slides

PIA Working Group Bi-Weekly Conf Call: Thu, May 28, 3p PT

(Zoom)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece, Justin Elser
• Todorovic Group (OSU, EECS): Xu Xu, Ryan Kitchen
• UCSB / Bisque Group: Kris Kvilekval

Agenda

• Software development progress
• Any ongoing issues with the local BisQue engine?

Action Items

• RK is working on the client-side interface
• XX is working on the local engine config and server-side Matlab graph cuts dev
• JP will set up new Planteome repo for PIA source code
• JP will set up new Planteome-Bisque dev mailing list

PIA Working Group Bi-Weekly Conf Call: Thu, Jun 11, 2p PT

(Zoom)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece
• Todorovic Group (OSU, EECS): Xu Xu, Ryan Kitchen
• UCSB / Bisque Group: Kris Kvilekval

Agenda

• New mailing list for this group: planteome-bisque-dev@planteome.org
• Software progress
  • Start using new Planteome repo: https://github.com/Planteome/planteome-image-annotator
    • Commit current work now, please
    • Need to get KK, DF added to GitHub team
  • Take a look at RK's laptop environment and UI work for our Bisque module
  • XX: progress on local engine config and server-side Matlab graph cuts development
• Dennis Stevenson @ NYBG: in situ images

News

• updated Bisque engine coming soon (5.8)...
• good example modules: NuclearDetector3D (latest/greatest), PHFDetector, NuclearFilter (reads Mex for input/pipelining)
• initial module code added to repo; initial matlab scripts coming shortly...

Action Items

• RK adding base module to new repo; adding background blue line markup to existing foreground red line markup
• XX finish engine config, add base Matlab graph cuts scripts to repo
• RK and XX create and confirm round-trip "ping" from client to server within Bisque dev module

PIA Working Group Bi-Weekly Conf Call: Thu, Jun 25, 3p PT

(Zoom)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece
• Todorovic Group (OSU, EECS): Xu Xu, Ryan Kitchen
• UCSB / Bisque Group:

Agenda

• Software progress
  • Update on Bisque module dev
    • adding background blue line markup to existing foreground red line markup
    • progress on server-side Matlab graph cuts development
    • do we have a successful client-server round-trip?
• Updated Bisque engine (5.8) ETA?
• KK, DF: Accept GitHub team invites, pls

Business

• XX: posted link to Kohli Dynamic Graph Cuts pub (our new algo for graph cuts module)
• RK: stubbed out round-trip from poly-lines to polygon

Action Items

• JP: talk to Nirav about iPlant server space (or explore using CGRB, UCSB BioDev, or EECS space)
• RK: get polygon to return from poly-line input
• XX: Build out Kohli Graph Cuts

PIA Working Group Bi-Weekly Conf Call: Tue, Jul 14, 3p PT

(Zoom)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece, Justin Elser
• Todorovic Group (OSU, EECS): Xu Xu, Ryan Kitchen

Agenda

• Software progress
  • Update on Bisque module dev
    • polygon returning from poly-line input?
    • Kohli Graph Cuts progress report
  • Updated Bisque engine (5.8) ETA?
  • KK, DF: Accept GitHub team invites
  • Dev staging space - BioDev seems best
• Dennis: Planteome-Bisque module and your images

Reports

• XX is working on using the existing GraphCutsSegmentation module as a starting codebase, and extending it to include dynamic graph cut functionality. NOTE: This would be a return to using the Kolmogorov algorithm, plus dynamic features. So long as performance is good and scales well, this model should be ok. XX will be conferring with Bisque reps.
• RK having trouble getting local Bisque engine to run Matlab modules; this is needed to test his "round-trip" from client to server and back. He will discuss with KK.

Action Items

• JP: talk to Kris about using UCSB BioDev as our Dev staging server (with build code coming from our GitHub repo)
• RK: troubleshoot local Bisque engine (specifically running Matlab) with KK; there could be an OSU licensing issue
• XX: Discuss dynamic graph cuts system design with DF: pure Matlab, or Matlab + C libraries?

PIA Working Group Bi-Weekly Conf Call: Tue, Aug 18, 3p PT

(Zoom)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece, Justin Elser
• Todorovic Group (OSU, EECS): Xu Xu, Ryan Kitchen
• UCSB / Bisque Group: Kris Kvilekval

Agenda

• Dev staging space discussion w/ KK
• KK, DF: Please accept GitHub team invites
• Software progress
  • Update on Bisque module dev
    • Client-server integration report: RK
    • Dynamic Graph Cuts progress report: XX

Reports

• XX has committed the Kohli DynamicGraphCutsSegmentation module to our Github repo, including a compiled algorithm; is currently working to understand interface so as to modify Bisque module to take graph cut output accordingly
  • confirmed that our implementation will be a hybrid of Matlab scripts and C++ libraries
  • Algorithm author (Kohli) also supplied the starting codebase via public download
• RK is now writing the server-side client interface portion of the module in Python, instead of Matlab
  • All agree that this is a good move for flexibility and sustainability
  • Discussed with KK: using MCR (Matlab Compiled Runtime) to get around OSU licensing problem; this is needed in order to round-trip test our module interface functionalities.
  • KK advised that we will need the OSU Matlab license on our Dev box (just make sure the runtime matches the compiler)
• KK demo'd registering a private Bisque module on their public Bisque engine
  • It was decided to set up our environment to do exactly this for integration and performance testing.

In sum, we will create a local Bisque engine (probably on dev.planteome.org) and migrate our centralized dev environment there (pulling from our GitHub repo). Development can still happen on local laptops, but it will also be possible to quickly code and deploy to this Dev engine. Then, we will register and,at intervals, update the version of our module for integration testing in UCSB's environment. Immediate goal: be able to view, run, and test module under development from both a local and a remote Bisque server.

Next step: Focus on a basic integration of graph cuts code and module code (user-facing and Bisque internals).

Action Items

• JP, JE, and RK: set up a local Bisque engine server; migrate module code to it
• RK: register module with UCSB's public server
• XX: continue mods on dynamic graph cuts code
• JP, RK, and XX: Meet to explore interface connections between graph cuts and user-facing module code

PIA Working Group Bi-Weekly Conf Call: Wed, Sep 9, 3p PT

(Zoom)

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece, Justin Elser
• Todorovic Group (OSU, EECS): Xu Xu, Ryan Kitchen, Sinisa Todorovic
• UCSB / Bisque Group: Kris Kvilekval, Dmitri Federov

Agenda

• Planteome Integrated Dev server update - JP, JE, RK
• KK, DF: Please accept GitHub team invite (this has been hanging out there for a while) - done
• Software progress
  • Update on Bisque module dev
  • Report on porting client-server piece to Python: RK
  • Report on Dynamic Graph Cuts progress (using ImageMatting now - pls explain): XX
• Discuss details of integrating server-side and client-side code: What are the interface points?

Action Items

• Complete deployment and config of Planteome Integrated Dev server - JP, JE, RK
• Define integration points between client and server software - RK, XX

Planteome Dev Group Weekly Meeting - 10 Feb 2016

Attendees

• Jaiswal Lab (OSU, BPP): Justin Preece, Justin Elser
• Todorovic Group (OSU, EECS): Xu Xu, Ryan Kitchen

Progress

• RK/DT/XX: all Dev servers running engine and client, and are returning results from existing registered modules
• group met with KK and debugged our Bisque test server; it is now running (although crashes occasionally)
  • main changes: Apache config proxy rewrites (ProxyPreserveHost); other Bisque config changes
• all developers have user accounts on the Bisque test server
• JP changed module folder name from TestModule2 to PlanteomeSegmenter
• SC: Ontology Service API: Term Search under develeopment (JP will follow up)

Issues

• Test Server stability: bisque-dev.planteome.org went down a couple of days ago; cause unknown; RK looking into it

Next Steps

• JP: confirm Term Search record format, begin designing ontology annotation Bisque UI and record format with KK
• RK/DT: client UI - fg/bg marks in JS override? (check with KK)
• JE/RK: git sparse to properly place our git module on palea; pls confirm round-trip commit back to github
• RK: re-brand all module files to PlanteomeSegmenter namespace
• XX: modify Matlab scripts to accept 1-to-many foreground and background polylines, and validate for at least one of each

Planteome Dev Group Weekly Meeting - 17 Feb 2016

Attendees

• Jaiswal Lab (OSU, BPP): JP
• Todorovic Group (OSU, EECS): XX, DT

Progress

• XX: Added code to module to identify 1-to-many fg/bg polylines; awaiting client mods to pass same to server
• RK/DT: investigating client-side UI mods to buttons and line-drawing events; need KK's input to progress at this point
• RK/XX: re-branded all module files to PlanteomeSegmenter namespace

Issues

• Test Server stability: bisque-dev.planteome.org still down
• Need KK's advice on client-side JS mods

Next Steps

• all: attempt round-trip with module in current state (only two markup lines accepted); when that works, deploy via git to palea and confirm roundtrip there
  • need to register module
• RK/DT: client UI - fg/bg marks in JS override? (need a meeting with KK - JP will schedule)
• JE/RK: git sparse to properly place our git module on palea; pls confirm round-trip commit back to github
• XX: validate for at least one of each FG/BG mark
• JP: confirm Term Search record format, begin designing ontology annotation Bisque UI and record format with KK

Planteome Dev Group Weekly Meeting - 24 Feb 2016

Attendees

• Jaiswal Lab (OSU, BPP): JP
• Todorovic Group (OSU, EECS): XX, DT, RK

Progress

• received JS framework walkthrough from DF; will be used to modify client viewer UI and behavior (buttons, line color, etc.)
• worked through some Matlab configuration issues on dev machines
• JP, KK met and began designing ontology annotation Bisque UI and record format with KK
• it was decided that the Bisque team will develop the ontology annotation enhancements in the Bisque application (our team will collaborate on specs and provide the ontology web service)
• Seth Carbon (AmiGO) has implemented an API for ontology term autocomplete and details (JE will get it running on dev.planteome.org)

Issues

• still having config issues with dev and test Bisque servers; this prevents modules from running
• Test Server stability: bisque-dev.planteome.org still down

Next Steps

• all: attempt round-trip with module in current state (only two markup lines accepted); when that works, deploy via git to palea and confirm roundtrip there
• RK/DT: client UI - fg/bg marks in JS override
• JE/RK: git sparse to properly place our git module on palea; pls confirm round-trip commit back to github
• XX: validate for at least one of each FG/BG
• JP: document ontology use-case and draft design for KK and Bisque group
• JE: pull down new AmiGO api extensions and run from dev,planteome.org

Planteome Dev Group Weekly Meeting - 2 Mar 2016

Attendees

• Jaiswal Lab (OSU, BPP): JP
• Todorovic Group (OSU, EECS): XX, DT, RK

Progress

• PlanteomeSegmenter module registered in local dev Bisque envs
• API running at dev.planteome.org. Examples:
  • term detail: http://dev.planteome.org/amigo/term/TO:0000050/json
  • term autocomplete: http://dev.planteome.org/api/autocomplete/ontology?q=leaf&fq=idspace:PO

Issues

• still having config issues with dev and test Bisque servers; this prevents modules from running: RESOLVING THIS IS NOW THE HIGHEST PRIORITY
• Test Server stability: bisque-dev.planteome.org still goes down occasionally
• API documentation page at dev.planteome.org/api is not serving (however, the API REST methods themselves work)

Next Steps

• all: attempt round-trip with module in current state (only two markup lines accepted); when that works, deploy via git to palea and confirm roundtrip there
• RK: Bisque engine config issues; get the modules running
• DT: client UI - fg/bg marks in JS override
• XX: modify Matlab code to expect two collections of polylines from the client (one fg, one bg)
• JP: document ontology use-case and draft design for KK and Bisque group; share API examples

Planteome Dev Group Weekly Meeting - 11 May 2016

Attendees

• Jaiswal Lab (OSU, BPP): JP
• Todorovic Group (OSU, EECS): XX, DT, RK

Progress

• round-trip with module in current state (only two markup lines accepted) works
• multi-line segment markup, with background/foreground toggle, is now possible in the client side

Issues

• still having config issues with dev and test Bisque servers; this prevents modules from running: RESOLVING THIS IS STILL THE HIGHEST PRIORITY, considering migrating to Bisque DOCKER instances
• Test Server stability: bisque-dev.planteome.org still goes down occasionally (would Docker solve this problem?)

Next Steps

• JP, KK: Bisque engine config issues; get the modules running - consider moving to Docker.
• DT/RK: client UI - foreground/background color differentiation
• XX: modify Matlab code to expect two collections of polylines from the client (one fg, one bg)
• JP: document ontology use-case and draft design for KK and Bisque group; share API examples

Planteome Dev Group Status Meeting - 23 June 2016

Attendees

• Jaiswal Lab (OSU, BPP): JP
• Todorovic Group (OSU, EECS): ST, DT, RK

Progress

Issues

Next Steps

• JP, KK: Bisque engine config issues; get the modules running - consider moving to Docker.
• DT/RK: client UI - foreground/background color differentiation
• XX: modify Matlab code to expect two collections of polylines from the client (one fg, one bg)
• JP: document ontology use-case and draft design for KK and Bisque group; share API examples

Planteome Dev Group Status Meeting - 14 July 2016

Attendees

• Jaiswal Lab (OSU, BPP): JP
• Todorovic Group (OSU, EECS): DT

Progress

• blue lines!

Issues

Next Steps

Planteome Dev Meeting - 13 Apr 2017

Attendees

• Jaiswal Lab (OSU, BPP): JP
• Todorovic Group (OSU, EECS): DT

DT Progress

• Desktop image classification tool: ready for internal testing and use to aggregate annotation data for machine learning adding bounding boxes
  • prepped and provided zip archive of test files for testers
• Machine learning: currently refining parametrics for back-propogation; no network convergence as of yet, but improvement using botanical training data sets from [ImageClef]
• Bisque: image classification prediction module is running on local Dev with a sample machine-learning network

Issues

Next Steps

• DT continues tuning convergence parameters on training set to improve predictive power
• adding bounding boxes and resulting annotation data to desktop tool

Conferences and Workshops

Please see: Planteome_Outreach