Beginning of the IfcOpenShell test environment development

J by johltn on 26 Sep 2021, edited 20 Jan 2022 # +10 votes

Dear OsArch community,

I’m Johan Luttun, a software developer specialized in BIM applications. I’ve always been interested in automating tasks for the AEC industry, as I believe there are still tools missing for a smooth and useful use of information models, at all phases of construction projects and besides it. I’ve worked on a wide variety of software projects, mainly with Thomas Krijnen from AECgeeks, on data validation and geometry.

I’m happy to be in charge of the implementation of the official IfcOpenShell test environment to offer a guarantee regarding the reliability of the IfcOpenShell geometry interpretation functions used by IfcConvert and BlenderBIM.

Objective of the project

The goal of the IfcOpenShell test environment is to provide a trustworthy source regarding the performance of IfcConvert geometric conversions - based on the underlying IfcOpenShell geometry interpretation - and make key metrics of those operations accessible to the users on the web.

To provide some context, we are not starting from a blank page as we already had some scripts in the pipes that now have to be tied together to make a real application out of them. The objective of the project is to reflect accurately and up to date the performance of IfcOpenShell regarding the conversion of IFC files to other geometric formats.

A part of the grant allocated to BlenderBIM by Epic has been dedicated to this project. Thanks to this support, I’ll be able to spend time to create the new scripts, tie them together, deploy the application and progressively implement a nice user interface.

The performance of the file conversion will be measured in terms of errors and processing times. The motive is to be able to answer the following questions after a new commit is made on IfcOpenShell:

  Does this commit make the conversions better or worse?

  How much does the performance change with this commit?

Of course, according to the users demands, this project is expandable, and I’d like to implement many functionalities over this year.

Overview of the application

The brain of the application, or the back end, will be implemented using the flask microframework and deployed using Docker and docker-compose. The application will be based on the ifc-pipeline open-source project https://github.com/AECgeeks/ifc-pipeline.

An overview of how the application will be working is available on the schema below. As you can see, the workflow relies on 2 main computations (steps 3 and 4).

  1. Query IfcOpenShell repository for new commits

The flask server will query the IfcOpenShell GitHub repository for new commits at a predefined rate.

  2. Check whether it is stored in the database

If the commit id is not registered in the database and if it concerns IfcOpenShell C++, it will save the new commit to the database, and launch the computations.

  3. Compute the IfcConvert builds

The server will build the IfcConvert executable for each OCCT version. So, there will be 2 builds per new commit.

  3.5. Query the test files repository https://github.com/IfcOpenShell/files to pull new files if necessary.

  4 . Compute the results

The computation involves IfcConvert conversions using the cartesian product of all the builds we have and all the files. Next, once the computations are completed, the logs will be parsed, and the results will be calculated and stored in the database.

Once the results are available, the front end can query them. A rough idea of how the application will look like is displayed below. I will use React to make things easier for the filtering and data visualization.

Next steps

As you may have guessed, the next months will consist in making a first implementation of the project, which will need the following steps in 2 parts:

Part 1

Create the missing scripts
Tie them together into a flask project
Set-up the server and deploy the application without or with very simple front-end

At this point, I’ll make the code available on Github.

Part 2

Once the server is set up, I need to:

Start to compute the builds
Request files
Compute the results

Then, I’ll implement the front-end, and the application will be available to users.

Once the application is up and running, I’d like to implement the following functionalities:

Test IfcOpenShell python recipes
Use new OSes for testing rather than only Linux
Allow users to have their own account with their own environments and files
Make the flask server as an API

I will try to post regularly about the Test environment on the OsArch forum and communicate on other media such as Twitter. Any interesting technicality will also be discussed on the IfcOpenShell blogs. Please feel free to reach out for any suggestion or question you may have about this project at johan@aecgeeks.com.

J by johltn on 20 Jan 2022 # +7 votes

Update about the development of the Test environment

<br>

The interface

I rewrote the interface in order to be more dynamic and visually appealing. The idea is to display cells that correspond to test results. On the screenshots you can see those cells for all the commits in the database (the end goal is to make it for all the IfcOpenShell commits on IfcConvert), the 3 latest commits, and one for an experiment i.e we could imagine that the users will have the possibility to run some tests with the builds they want on their own files.

The results are from old tests, which were stored in a resulting flat file, and that I stored in a database. Then, when the page is requested, the data is retrieved from the database and displayed in the tables and in the charts.

When the deployment is done, the 2 boxes before the cells will be updated when a new commit on IfcConvert is made. The first box will show the progress of the building of IfcConvert (with a donut for example), and the second one will show the progress for each file with the completion of bars.

The code is based on ifc-pipeline and available at https://github.com/johltn/ifc-pipeline/tree/testenv

<br>

Some improvements to be done

I think in the future it would be preferable to fetch the data continuously to feed the tables and charts, rather than passing it from flask when the page is requested. The stats about IfcOpenShell codebase (highlighted in green) are not dynamic for now but it won’t be difficult to change that.

Currently the interface is not responsive.

Deployment

The next big step of the project is the deployment, and the issue that we need to adress is how to store the executable IfcConvert files? There will be hundreds of them at least, and which provider to use? So that it doesn’t cost a fortune to store and to run the conversions for the historical commits and for each new commit. I will discuss it more deeply with @aothms who has some ideas about it.

D by duncan on 20 Jan 2022 #

I barely understand what you're doing - but it looks great!

J by johltn on 20 Jan 2022, edited 20 Jan 2022 # +2 votes

@duncan said:

I barely understand what you're doing - but it looks great!

Thanks! This work aims at creating an interface to display the results of tests that consist in conversions of a set of IFC files using different versions of IfcConvert http://ifcopenshell.org/ifcconvert and for different versions of OCCT. For each test we're storing the conversion execution time and the exit value. And this also involves all the back-end work to continuously build and store a new version of IfcConvert when there is a new commit, launch the execution of the tests, and store the results. So basically this project aims to test the performance of IfcConvert and IfcOpenShell libraries as they power it :)

M by Moult on 20 Jan 2022 # +1 votes

@johltn this is great work! By the way I have a dedicated Gentoo server which runs the OSArch infra, and it's got a 2TB drive, if that's of any use.