IfcOpenShell - project and repo structure - modernise/split up/clean up

D by doia on 25 Aug 2023, edited 25 May 2024 # +1 votes

I would like to start a discussion about the project/repo/folder structure of the IfcOpenShell project and possible ways to modernise and maintain a concise project structure.

Currently the repo is a big bag of everything. Literally. The main parts are IfcOpenShell and BlenderBim, both massive projects on its own. And there are more smaller projects which depend on IfcOS, but are not necessarilly a vital part of the core project. More like smaller utilities or additions, even just explorations into other areas. Some source code is even unmaintained or stale.

I would like to propose a split of this monorepo into smaller units. I think this would greatly help in the general work on both projects and make these parts better maintainable.

finding your way around the project is challenging. There is so much going on right now that is is difficult to grasp the interconnectivity between the single parts
most issues and commits happen in regards to BlenderBim. It is hard to keep up with changes to IfcOS-Core or other parts of the project
The branching structure is very unusual. The main work happens in branch v0.7.0 (SemVer!) not in a main or master branch, although these exist too
The folder structure is odd. Everything lives in /src, even if it’s just examples or tools. Dead folders are kept around, even though they are preserved in the git history.

What do you think? Do you (the project maintainers) want to touch this topic and are you open for PR’s?

M by Moult on 25 Aug 2023 # +1 votes

Just about to go to sleep, but this has been brought up a couple years ago here: https://github.com/IfcOpenShell/IfcOpenShell/issues/1497

D by doia on 26 Aug 2023 #

Thanks for the link. I think I have seen this issue before. Yet I think it is time to revive the discussion.

Your (and @aothms) points are mostly about:

the tight connection between IfcOpenshell and BlenderBim, not by code coupling but as GUI complement to a core layer and in lockstep of development
the convenience of developing BlenderBim together with IfcOS in a shared repo vs. separately compiled

These arguments make it seem like IfcOS as a library only exists for the benefit of BlenderBim. It leaves users with other use cases and projects out.

Ones which do not need BlenderBim as a GUI but use IfcOS as a dependency on their own projects or in other context, lets say within a Jupyter notebook or a small stand alone utility program. Or even an alternative desktop authoring app based on IFC. Sure, I can use IfcOS as a dependency, but I also get a lot of excess luggage too.

I’m not so really concerned about the notification noise within the repo, as mentioned in the issue. I would rather argue in the context of IfcOS as an independent library, just as it says on the tin (ifcopenshell.org). With feature parity of the C++ and the Python API.

One thing of the tight dev workflow between both projects is an incredible advancement in the Python API. But I have the feeling the the C++ API lacks behind.

So in short, I would favour having a IfcOS library which does one thing (or a well defined set) and does it well. I think this would also benefit the spreading of IfcOS within the wider aec eco system.

Side note:

Recently there was an interesting article about the core philosophy of the Unix program design. Libraries and apps, which do on thing and do it well.

The IfcOS repo currently IMV is not well defined. It looks more like a huge playground, where everything is thrown in. This is not beneficial for longtime maintenance.

Sometimes you need to prune a tree to encourage growth.

D by doia on 26 Aug 2023 #

Here are some notes on the IfcOS repo I made while exploring the code base. These might give an outside view of a newcomer to the source base and trying to understanding it as opposed to someone who is intimate familiar with it.

notes on the root folder structure:

there are several subfolders for different build systems directly under root, like cmake, conda, nix etc. -> these folders should live under a common /tools folder to make the use case clearer
most of the subfolders do not have a basic README file to help understanding the content and usage
/choco/blenderbim - tools to build the BlenderBim add-on as a chocolates package
/cmake - main CMakeLists.txt file to build the IfcOS library with 1000+ LOC -> split up into smaller modules?
/conda - build setup files to use Conda
/docs - build the documentation using Sphinx and Doxygen, seems to be outdated, as some links within the Spinx setup point to an obsolete src/ifcblenderexport folder
/nix - build setup files to use NixOS
/pyodide - build setup to use the Pyodide Python distribution for use with Node.js and WebAssembly
/test - downloads a lot of external .ifc and .zip files and perform tests on them, but partially also relies on external Blender installed
/win - windows build tools and scripts for use with Visual Studio

notes on the /src folder:

I consider bold marked folder names as belonging to IfcOS core functionality

/bcf - simple Python implementation to handle bcf-xml files, has a dependency on ifcopenshell (Python API), but in my view not necessary for the core functionality of IfcOS
/blenderbim - the Blender add-on to use IFC -> should be a separate repo and use IfcOpenShell-core as a dependency
/bsdd - experimental wip library to interact with the buildingSMART Data Dictionary (bSDD) API, last commit
examples - C++ examples using IfcOS -> the folder should be located under /root, most files lack an explanation or comments on what they do
/ifc2ca - use Code Aster in IFC-driven FEM analysis -> not a core functionality of IfcOS, rather an extension to use the IfcOS Python bindings with Code Aster
/ifc4d - utilities for converting to and from various 4 D software, uses the IfcOS python bindings
/ifc5d - utilities to import or export cost-related date from ifc files
/ifcbimtester - utility to test if an ifc files satisfies certain requirements, actually a standalone app and add-on
/ifcblender - legacy Blender import add-in -> should moved into a separate repositiory aligned with the BlenderBim source
/ifccityjson - utility converter CityJSON files to IFC
/ifcclash - utility for IFC-based clash detection
/ifccobie - utility to extract COBie data from IFC to spreadsheets
/ifcconvert - single C++ file to convert geometry from an IFC file to other tesselated and topological formats, basically a file converter, 1600+ LOC, no README
/ifccsv - utility to interact with IFC data through CSV
/ifcdiff - utility to compare IFC files
/ifcfm - prototype utility to supersede ifccobie, extract and analyse IFC data for the purpose of facility management
ifcgeom - core C++ files for IFC geometric types, converting IFC schema types to Opencascade geometry types? Needs investigation!
ifcgeom_schema_agnostic - ???
/ifcgeomserver - C++ example to expose IfcOS API through a cli-based stdin interface, 650+ LOC -> not part of the core library rather an external app
/ifcjni - JNI (Java Native Interface) to IfcOS for use with BIMserver.org
/ifcmax - outdated legacy extension for IFC support in 3DS Max -> just as the BlenderBim add-on an external project
/ifcopenshell-python - Python bindings and high-level API for IfcOS
- /ifcopenshell
  - /api - python high-level functionality build on top of IfcOS C++ base functionality -> is this functionality also available from C++? If not, why not?
  - /express - this seems to be the source folder to extract C++ files from the IFC Express schema -> but why burried down into such a deep subfolder if it is integral to create the IfcOS functionality? -> this should be moved under a /tools folder under /root, as this is used to create IfcOS C++ code, but not for the actual use of IfcOS
  - /mvd - mvdXML checker and w3c SPARQL converter, as an IfcOpenShell submodule or stand-alone
/ifcparse - not yet explored, missing README
/ifcpatch - not yet explored, missing README
/ifcsverchok - not yet explored-> this seems to be yet another Blender Add-On
/ifctester - not yet explored
/ifcwrap - Python wrapper around the IfcOS C++ core build using SWIG
/opencdeserver - ???, as far as I can tell from the source code this folder has nothing to do with the IfcOpenShell project -> why is it in /src?
/qtviewer - an old sample program to display IFC files, last change 11 years ago, updated to use QT5 1 year ago -> should be a separate repo IfcOpenShell-Viewer with a dependency to IfcOpenShell-core, or at least under a /root/viewer directory
/serializers - lots of serializers into/from different formats, huge files with a lot of LOCs, missing Readme.md with use-case
/svgfill - a git submodule to https://github.com/IfcOpenShell/svgfill -> this should be located under a /root/thirdparty or /root/external folder to make the dependency relation clear
parts of the c++ code are not build as libraries, they are included via #include "../ifcgeom/IfcGeom.h", but should be included as #include "ifcgeom/IfcGeom.h" to make them path independent

M by Moult on 26 Aug 2023, edited 27 Aug 2023 # +5 votes

There seems to be a few comments here. Let's start with the things I 100% agree upon:

IfcOpenShell doesn't use semver. I think that's a con, especially for a library, and ideally should be reevaluated. I like semver, but I don't think it applies to alpha software which BBIM is, which is why I use dates in the 0.0.X stage. When we go into beta, i.e. 0.X.X, then semver will be more closely adhered to.
Totally agree that the branching structure is unusual. This ties a bit into the semver things. Apart from the big "0.7.0" release, everything on top is treated as a rolling release. This is not sustainable once things start to mature.
The folder structure definitely is odd. It's definitely confusing when there are folders that mix C++ and Python. Rethinking the folder structure sounds like a good idea.

I'd like to clarify a few things:

Historically, IfcOpenShell began as a collection of tools. One of those tools was "ifcblender". There then was an "ifcblenderexport", which then merged to become "blenderbim".
Just because many things are in one repo doesn't mean that they don't individually "do one thing and do it well". I think that each tool does have a pretty well define scope.
Missing READMEs are because those are slowly being migrated to Sphinx instead. See https://blenderbim.org/docs-python/ where you will find a lot of those READMEs.
Unmaintained or stale code seems to be an irrelevant point. This occurs regardless of branching, monorepo vs splitrepo, semver, or folder structure. Probably split repos might even make more code stale as "out of sight, out of mind". This is more a factor of developer resources.

I'd also like to emphasize a few points of view which I've come to really appreciate over the past few years:

We get a lot of bugreports. The BlenderBIM Add-on is a huge source of these bug reports, but a number of these highlight more fundamental issues in other utilities or IfcOpenShell-Core. Rather than having multiple issue trackers spread across multiple repos, I really do like having a central issue tracker as bugs often get reassigned or apply to multiple tools simultaneously. The reporters may not know which tool their bug actually applies to. Once we triage it, we can assign the appropriate tool.
The BlenderBIM Add-on plays a huge role in being a de facto UI for many of the other tools (IfcCSV, IfcPatch, IfcDiff, IfcClash, IfcOpenShell-Python...). So The development really is quite intertwined. If a new feature comes out in IfcCSV, it often leads to a revised UI in the BlenderBIM Add-on, and perhaps improvements to the utils module of IfcOpenShell-Python. Similarly, a lot of features in SVG serialisation is based on development in the BlenderBIM Add-on. From the perspective of users who need graphical interfaces, they really are the same project.
That issues came up 2 years ago. In those 2 years, I've yet to think once that "oh, this'll be better if we had split repos". Our versioning is tied together, there are lots of spillover effects from each utility, it makes it easy for bugreporters to know where to report, we don't need to monitor commit access / issue trackers / CI workflows / across many more repos, and the most important thing: I don't want to commit to three different repos just to build a feature (e.g. recently I did a bunch of work on IfcCSV. This involves commits simultaneously to IfcOpenShell-Python, IfcCSV, and BBIM). Splitting the repos (let's say a minimum of three repos: IfcOpenShell Core+Python, BBIM, and "Other Utilities") is really going to hurt developer productivity.

I don't see how splitting repos helps users who only use IfcConvert, or only use IfcOpenShell-Python. We already bundle and package them separately, so isn't that sufficient?

M by Moult on 26 Aug 2023, edited 27 Aug 2023 #

Might I propose some low risk smaller scale changes that should have an immediate benefit and address some of your concerns first?

Finish writing the equivalent of a readme for all utilities in the Sphinx docs. At least then there is an intro for everything.
Help document recent versions of ifcmax which are compatible with 3DS Max 2023, and ifcblender which is still incredibly valuable as a minimal add-on. I don't think there is any actual obsolete / dead code in the repo, although it might seem like it, it actually isn't.
Finish deprecating ifccobie and merge into ifcfm.
Merge https://github.com/IfcOpenShell/IfcOpenShell/pull/3574 which updates qtviewer to Qt6

That's probably the easy stuff. I'd then perhaps propose to start reorganizing non source code, stuff like build scripts, examples, docs.

Finally, the hard stuff: actually moving source, improving branching, adopting semver, etc.

If this sounds like a practical plan, would you be interested to help contribute? Especially in the C++ side as you can see we are really short on resources in improving docs and examples. There's also a lot of work done in the v0.8.0 branch and refactoring in progress.

D by doia on 27 Aug 2023 #

Thank you for that extensive explanation, very insightful.

I see from the project history that a lot of the structure makes sense, especially in light of its organically growth. Even if I don't agree with everything from a current view onto the project, I will leave it at that. Yet I would appreciate if the project stays open to continuously challenge its status in terms of structure and organisation. I certainly will nag you periodically.

There's also a lot of work done in the v0.8.0 branch and refactoring in progress.

This quote highlights my confusion with the unusual branching model very well. The first look into this branch says "This branch is 332 commits ahead, 1380 commits behind v0.7.0.". Branch v0.7.0 seems current, branch v0.8.0 might be in dev but the git status and commit history tells otherwise. So how would I know?

I like your proposed steps for a refactoring in small steps. Let's start with that, I'm happy to support in ways I'm able to. Some remarks though:

Readme

I think you talk about the extensive documentation of modules. This is very important, especially in terms of deeper explanations of internals and architectural decisions, examples, use-cases etc.

But if I explore the code base in an editor and have a subfolder open (e.g. /ifcconvert or /ifcdiff), I would have to open an external browser to the docs or change folder to ../ifcopenshell-python/docs/ifcconvert.rst. Not very intuitive.

I talk about a short and concise Readme.md to give a quick overview of the contents and purpose of a folder/module so I can find my way around. Good examples are the the Readme files in /ifc2ca or /ifcgeomserver.

would you be interested to help contribute?

Yes indeed, I do. In fact I already tried, but the response so far was rather poor -> https://github.com/IfcOpenShell/IfcOpenShell/pull/2393

M by Moult on 25 May 2024 # +1 votes

A little update on this that the 4 easy proposals are now implemented. The next thing on the list is to adopt some form of semver, better versioning and improved distribution.