I Have A Dream: Code Visualization
There are some things I find absolutely addicting. Besides of code beauty and clean code and education, which I find interesting, I am also a great fan of gamification and visualization.
I wanted to find out how gamification might work in the ABAP programming environment and therefor started the GLADIUS project where you could kind of code-fight ABAP problems.
Visualization in the SAPGUI is not very easy because the graphical conditions are quite limited. There are UI5 components which can be used better for visualizing data and figures, like this UI5 demo. Some visualizations are also possible in SAPGUI but often very complicated.
There is a very interesting aspect of visualization and some special data: source code. Some people also tried to visualize source code dependencies or specials aspects of code like lines of codes, number of methods or attributes of a class and many more.
I will try to give an overview on existing projects and ideas that I have… Sorry for this long article. I figured out to split into mutliple posts but decided that one larger article might be better. Have fun reading!
Before you can visualize source code, you need to have some metrics that you can apply on the displayed objects. Because there are limitations on the amount of different metrics and displayable data for one object, you will have to define some views depending on your intent.
One of the most metrics used might be LOC – Lines of Code. This metric is very simple and yet useful in similar cases. Other metrics might be
- Number of comments per lines of code
- Number of methods of a class
- Number of attributes of a class
- Number of dependencies (where-used list)
- Code coverage
- Number of changes
- Number of contributors
These metrics can be more or less easily be found out. There are other metrics that are harder to determine and sometimes are not directly stored with the code, like:
- security issues
- number of issues
- number of short dumps caused by this function
- number of transactions
You might also consider to add beauty metrics:
- pretty printer executed?
- indented code
- number of spare lines
I know that clean code and beauty depends on personal views but some might be quite countable if you run abapOpenChecks by Lars Hvam on it…
There is a good overview of code metrics by ndepend.
Using Metrics for Visualization
If you want to visualize code you have limited options because you can only use a few of the existing metrics to build an object. If you use three metrics for the width and length, the height and the color then you are done. Other metrics can only be displayed by tags or additional tags but not really visualized. Of course you can have something like a dashboard to display all the metrics but that is not what I understand by visualization.
At the end of this post I will try to tell about some ideas of how to display more metrics at once.
Code Visualization Projects
Here are the projects and examples that I found on my search for code visualization. There are many projects that seem to be quite active although the demo sites are down. Some projects are some years old and do not seem to be maintained any more.
The first time I got in contact with code visualizations was with Moose and the contributions of Rainer Winkler to it. Moose provides a some very nice pictures of code like the one on the main page of Moosetechnology.org:
Moose can be used to view the source code parts, get an overview on the number of objects and its dependencies. The more lines of code an object has the bigger it is. Private methods are red and protected are blue (or vice versa). Black dots are attributes. Or something like that. Sorry, I don’t remember exactly. Dependencies are shown by lines.
Rainer Winkler wrote some blog posts about it and uses it for his daily work to understand and document legacy code.
Another great but maybe discontinued project is Code City by Richard Wettel. Code City visualizes source code in form of a city. Each class is a block and each method a building. The more lines a method has the higher the building is.
Code City also uses Moose as platform.
Promising: there is a Codecity Eclipse-Plugin in the Eclipse marketplace. I assume that it will not work with ABAP…
The Code is Beautiful project of Quantified Code provides some very promising screenshots:
Some source code is on github but the demos are not reachable.
5 Tips to help you visualize Code
On this web page some code visualization tools are named and provides some further links.
Code Radar is a project to visualize code similar to Code City and also provides a functionality to compare two different versions of a code base which you can explore in the demo.
Softvis3d by sonarcube
Also very interesting is the project Softvis3d. Also looks very promising but same here: unluckily the some demos are not running althouhg the github project seems to be quite active. Try out this demo.
The company ndepend offers a complete suite for code metrics, code visualization and more.
The above projects are great and can be really helpful to understand and document code. But after my son got an Oculus Quest 2 VR headset on Christmas day, I have the dream to explore code in virtual reality. The Headset and the experience within this virtual reality is so amazing that I can think of many things and options to do there. One thing is to walk through classes and into the code. And at least I am not the only one who was thinking of it…
The ICE or ISUE Lab in Orlando, Florida, already did some researches on this topic.
They built a world where code is displayed as a 3D world that can be explored by walking from class to class.
Inside each class you can see the code of it:
Again there is very little information about this project. The website shows that there are a lot of things going on, but nothing about the Code Park project. There is a video where I took the screenshots from.
There is one abstract I found by the Aalen University where the idea of 3D-code visualization and virtual interaction has been described very detailed. The screenshots I see here come quite close to the main idea that I had and that I will describe in the next chapter. The parts of the code are displayed as real buildings with structure and windows.
VR Code City
I wrote a lot about existing projects, metrics and I hope you got an idea of what could be.
Now I would like to give you some idea of how code visualization, virtual reality and gamification might come together…
Up to now we learned, that buildings are a good idea to visualize code. The last chapter comes quite near to my idea of how a I imagine a real cool 3D source code world. The buildings are not only blocks but buildings that look like buildings with windows and some architecture. At least they have only four dimensions:
If you walk through this city then there could be code of arms or emblems of all contributors of this class. The person that last changed the object could be highlighted by a fire ring or purple glowing mist.
building types and attributes
Objects that someone is working on can have a scaffold around it.
Classes that haven’t been changed for years and do not have clean code might look like this:
If there are a lot of changes and abapOpenChecks has no “errors” then the building could look like this:
The programs behind the most or common used transaction could be placed in the middle of the city or directly at the harbour so it can be accessed easily:
Buildings can have attributes that clearly show some metrics:
- A test coverage of 100% is sourrounded by green mist while objects w/o unit test are on fire.
- Open issues for an object can be visualized by a shiny red laser beam pointing to the sky.
- The more often an object has been changed the wider and better the way to it. If the object hasn’t been changed for years then you might need a machete to get into the house.
If you can visualize separate programs, transactions and classes, then it might also be useful to display other objects like dictionary tables or maintenance views. Tables could be displayed as tanks or storage houses. The more data in it the bigger the storage.
Periodic jobs could be buses, trains or horse carts driving through the streets. Also processes could be visualized by moving objects. Although I have no idea of how to retrieve the data for it…Modules might be districts of the city.
Pictures taken from game Anno 1404 with help of http://annowelt.eu/
VR code fights
As mentioned in the beginning, the GLADIUS project wants to make the programmer fight for better coding. What, if you could fight the Code Inspector monster by changing as many CI findings in a given amount of time. You could challenge your colleagues at the end of the day to find the worst problem or the most important issue. In half an hour you will have to fight as many issues or findings as possible. At the end of the day you made a building better, more beautiful then it was before.
The code inspector findings of a class could be a monster with as many arms as there were findings. And if I talk about monsters, there is one ABAP programmer who is specialized in monster who might have fun to build them in 3D: Paul Hardy your turn… 😉
At first glance all of this might be too much effort for having fun at work. But I am convinced, that some good visualizations, metrics and game rules can provide much better and stable code. This 3D Fylthrought might be a big overload but, hey, this is a dream. 😉 And modelling and interaction is not that complicated using the unreal engine or unity. The art will be to build houses dynamically and nevertheless beautiful. Plus showing relevant information and visualize changes when code was improved. VR and coding should also be possible as there are at least three different virtual desktop apps which make it possible to view the PC desktops in a virtual environment.
Of course no one would spend so much effort to design and build such a fantasy landscape of an ERP code base… Or maybe some day?
Damir Majer had this dream already some years ago. Let’s see if this dream will come true someday…
Enno Wulff Great to collect everything in one place…
you can add this one into your blog for above list as collective..
(Zoom in and out and move your mouse)
Very nice ideas! I've been working on a dependency map for ABAP (perhaps one day to be open sourced). So I have had a little insight on the complex interdependencies of SAP objects.
But classes, methods and tables are really just the tip of the iceberg, there are so many other object types in the ABAP universe, it gets really messy - and fun - the deeper you dig.
Thanks Mike Pokraka for your comment! Yes, source code is only a part of a lot of other objects, but in my opinion the objects that are changed most. A database table is at it is, there is no need of optimization or beautyfying. It's good to know that it's there but no need to have a gamified view.
Ah but depending on point of view they are very relevant. If two programs share a data element then they share a dependency. I often find connections via tables and structures.
Or consider the question of where a Z-Structures is used? In a SAP Standard transaction perhaps? Easy to spot if your visualization shows a dependency chain Transaction > BAdI > Implementation > Class > Method > Structure.
Or consider something as benign as a method calling a function module. If the FM is self contained then good, but if it uses global data of the function group then there is a dependency to the function group and all other FMs in it. And then you get the mess that internally it has a dual nature of being both Function Group/Module objects, but they also consist of a bunch of includes.
All fun and games 🙂
Depending on the point of view is the relevant phrase... 😉
I was more source code triggered. But yes, it would be cool to directly see the dependencies from within the "building". Maybe by entering the utility room or looking into the garden? 🙂
There are many metaphors to choose from. To debug a program we have to follow a little robot as he visits all the houses (classes) and goes into each room (method), carrying bits of data around.
In my opinion, code visualization is absolutely necessary in the future. Probably more source code is created every day than there are developers maintaining it. At least if the source code has been in use for years. Everything that helps with maintenance will be necessary!
Hey Michael Keller, Code Visualization is only one part for maintenance. If you want to really work with it, then there is a lot more to do. Rainer Winkler can tell about it... There must be an out-of-the-box solution for the developer. If it is hard and complicated to setup, then it will not be used. For example with Moose, the following steps are needed (as far as I remember):
It would be far better if there was an eclipse-plugin...
I agree with you. The mere existence of a possible solution doesn't mean that it is easily accessible. But that is precisely what makes up part of the value in everyday work!
Even though I find documentation very important, the documentation I wrote for SAP2Moose and Moose2Model is not as good as I would like to have it. I find that it motivates me when a user open an issue in Github to describe problems with installing and using a tool.
Writing complex open source applications requires a lot of unpaid work. The main motivation is, that I need these applications for my daily work. But the documentation I need for myself is not the kind of documentation other users need.
As users get the application for free, I find it OK, when they invest some time, to give feedback with Github issues and explain where they need more documentation. This requires much less effort compared to the effort to develop and maintain the application itself. They may also fork the repository, improve the documentation and add a pull request. 🙂
With a complex open source project you need support. There are many opportunities for supporters.
On the subject of "unpaid work" I have also been thinking a lot lately. Often, as unpaid work, you do topics that you enjoy and interest yourself. That often only fits after paid work. But of course there is still family and other obligations like a house 🙂 A difficult and complex subject.
With proper documentation the steps Enno Wulff describes are not as difficult as it may look like. I agree that a plug-in in Eclipse would be nice, but I have no resources to write a third big and complex open source project.
When I use Moose for my daily work, I work with Moose2Model. This tool makes it possible to create something similar to conventional electrical circuit diagrams; I propose to call it circuit diagram for software. I have many such diagrams in my customer projects. Whenever I start a topic or come back to a topic, I search for an existing diagram or create a new one. These diagrams stay near to the same for long time, but it is still very simple to adapt them when the application changed.
In these diagrams, I see most dependencies of the code I am working in. I jump to ABAP Eclipse or the BW modeling tools in Eclipse to display or edit the code. Because SAP contains much more language features, than just ABAP, I enhance SAP2Moose from time to time. Because everything is mapped to package, class, method, and attribute, Moose2Model is able to display everything which is extracted by SAP2Moose.
I think FORM routines and old DYNPRO programs should come out as medieval buildings. They can be beautiful, but most are not. I shudder to think what SAPMV45A would come out looking like. Gormenghast falling to bits and on fire I suppose.
OO Classes should come out as science fiction thingies like starships.
Monsters of course don't care either way. They can crop up in the old fashioned buildings and in an ultra-modern starships. In fact the analogy is good because a monster can do a lot more damage in a spaceship than in a thatched wooden hut i.e. poorly designed OO code can be a lot worse than poorly designed procedural code.
Looking at the "Moose" picture which yields a sort of circular image as a result it seems that a really bad program would come out looking like the Coronavirus.
Hey Paul Hardy I think that SAPMV45A would be a 278 floors dark and musty dungeon below the city... 😀
Hey Paul Hardy I found a picture of Sapemveefortyfiveay Beach:
Another interesting approach could be to detect HotSpots of change using the history in Git. I am currently reading the book Software Design X-Rays: Fix Technical Debt with Behavioral Code Analysis by Adam Tornhill. There is also a video on Prioritizing Technical Debt as if Time and Money Matters. The tactics for identifying focus areas for improving legacy code look very promising. There are also existing tools e.g. CodeScene or code-maat.
Looks interesting, Klaus Haeuptle! Thanks for this input.
One part is to gather the data, however this data might look like: objects, source code, dependencies, beauty or as you mentioned Hotspots.
The second part is the visualization itself. as described, you will need to have different views to visualize the data. A hotspot might be an additional attribute (color, badge, height, ...) of a building.Or maybe a filter. The hotter the spot the more the building will be on fire or the more money will fly out of the windows when walking through the city with the "technical dept" glasses on.
Thanks for the link to Prioritizing Technical Debt as if Time and Money Matters. I found that very interesting!
In a classical SAP customer system, one could identify the hotspots by analyzing the transports (bit tricky, with R3TR and LIMU, and you need to parse the code to identify the changed procedures, but possible).
And then apply focused refactoring for those objects/procedures like:
thank you for this very inspiring post. I was really exited about the idea of visualising our code the way code city does. So I extracted a model using abap2famix and tried to display in it within code city. Unfortunately this was not successful.
After a while I found CodeCharta (MaibornWolff/codecharta: CodeCharta visualizes multiple code metrics using 3D tree maps. (github.com)) which comes with extractors for several languages and also allows to visualise project in any language easily by offering a simple json import.
So I used the metrics from transaction /SDF/CD_CCA, enhanced them a little and wrote an ABAP report to convert it to CodaChartas json format.
Code City from ABAP Project
In this example i used number of statements per class for the building height, the average number of statements per methods for the floor space (clean code tells us that it should be around 3 to 5) and the number of changes for the colour.
/SDF/CD_CCA offers also other metrics like complexity or decision depth.
Playing around with these metrics seems like an easy way to identify those objects where a refactoring promises to be most beneficial, because large and complex objects that are changed often tend to introduce bugs and slow down the development process.
It also seems that this visual approach helps to promote topics like clean code within an organisation as the negative impact of such skyscraper classes becomes clearer when you look at these graphics.
I think I will take a Code City Snapshot from time to time to track how the red skyscrapers steadily turn into beautiful, clean and green suburbs.
Even without medieval buildings that can be accessed using VR, this might help a lot to navigate towards a clean code base.
Hey Benjamin Weisheit ! This is really great news! Thanks for presenting this project. I wonder why I didn't find it...?!
CodeCharta really looks very nice and seems to have completely run out of proof-of-concept-phase.
How did you use the Custom Code Analysis transaction to provide applicable metrics for CodeCharta? Would you mind presenting your report to github?
CodeCharta somehow interacts or uses SonarQube plugins or basics. Maybe I will find out how they come together...
btw: I also found another working city-building website by Uni-Leipzig: Getavis
the city metaphor City Bricks looks very interesting!
Hi Enno Wulff, sorry for the late reply. I really hope to be able to share the source code via github but I need to check if this is compliant to my companys policies first. I will share to code as soon, as I have the permission, sorry 🙁 .
So for now, here's only its logic:
Getavis looks really useful. I took a look at Uni-Leipzig's site and I think I saw that they already visualised ABAP code. Did you try it out or are you in contact with them?
Thanks for this further detailed information, Benjamin Weisheit !
I did contact some persons but not yet the responsible persons.
Here's the code:
BenjaminWeisheit/ABAP-2-CODE-CHARTA: Covert ABAP Metrics to Code Charta (github.com)
Depending on your requirements you may want to adjust the logic for the aggregation of the results from /sdf/cd_custom_code_metric. This is implemented in method zcl_i_metrics_2_json->aggregate_metrics. I would be curious if you or someone else has any ideas, how this part could be improved.
E.g. Should Functions groups be treated like classes or is the approach to simply sum up the complexity of all methods within a class.
That's great! Thanks a lot! I will check ASAP
This is so cool!
Welcome to AbapGit City
amazing work. I am about to clone it and run the metrics in our system. I was going to answer your e-mail but then I thought I also should some credits here.
Usage & Dependencies would be nice Information too. As for the latter one, I cannot get TRON out of my mind :-). Also Dumps in the system to let the city burn 🙂
We need definitely talk about all that.
In the meanwhile have fun and take care.
An interesting approach could be to detect Hot-spots of change using the history in Git. Can try it. I m m trying to build an AI based system to work on it.