Friday, December 01, 2017

Revisiting Continuous Integration with PowerBuilder 2017, Bonobo Git and Jenkins

In a previous blog post I ran through a example of how to use the new command line compiler feature of PowerBuilder 2017 to perform continuous integration using Jenkins and a Git repository.  One of the "next steps" I referenced in the close of that post was the creation of a plug-in for Jenkins that would make creating the build steps for the project much simpler.

Well, I've created such a plugin.  Or more specifically what I did was take the existing MSBuild plugin and customize it to create a number of plugins to do PowerBuilder compiles instead.





In this post I'm going to look at how we can use those plugins, plus some additional changes I've made to the project in Jenkins to make better use of the capability of Jenkins and other Jenkin plugins.

Download and install the Oracscript and PBCompile plugins


The first thing you'll need to do is grab a copy of the OrcaScript and PBCompile plugins.  What you'll need is the orcascript.hpi and pbcompile.hpi files.

Go into Jenkins and select the "Manage Jenkins" option, and then the "Manage Plugins" link on that page.


On the Plugin Manager page select "Advanced" and then look for the "Upload Plugin" section. Click the "Browse" button there and navigate one of the HPI files you downloaded.



Do it again for the second HPI file.  You'll need to restart the Jenkins server after installing the plugins before you can use them.

Configure the plugin installations


After the server has been restarted go back into "Manage Jenkins" and then "Global Tool Configuration".


Scroll down that page an you'll see a new "Orcascript" section.  Select the "Add OrcaScript" button.


In the form that appears, provide the installation with a name and the path to the OrcaScript executable (orcascript170.exe).




Scroll down the page a bit more and you will see a new "PBCompile" section.  Select the "Add PBCompile" button as before.  In the form that appears,provide the installation with a name and the path to the PowerBuilder command line compiler (pbc170.exe).



Create OrcaScript and PBCompile build steps


Go back into your PowerBuilder project and select the "Configure" link.




Delete all the previous build steps that called batch files.  Now add a new one and notice a new entitled "Create PowerBuilder PBLs from source code".  Select that one.


In the form that appears, select the version of the OrcaScript installation you just created and provide the relative location to the PowerBuilder project file.  If you have PBDs in the library path that you want OrcaScript to ignore, provide them in the Exclude Liblist option.  The open source PFC contains a reference to the PBDOM PBD, so we're excluding that here.


Add another build step and this time select the "Build a PowerBuilder target using PBC" option.


In the form that appears, select the version of the PBCompile that you just created earlier.



You'll also need to add the command line arguments to pass to the PowerBuilder Autocompile utility.  You can get those from the first page of the project object in your PowerBuilder target.



These plugins provide the same capability as the first two batch files from the previous blog article, namely running OrcaScript to create PBLs from source control and then running the AutoCompile Utility (PBC170.exe) to create the executables and PBD from the PBLs.

Archive the Project Artifacts


What we're going to do now is replace the third batch file, the one that copies the EXE and PBDs into the build folder.  Jenkins has some built in capabilities that make this simpler that I didn't demonstrate in the previous article.

The first thing we're going to do is to tell Jenkins to archive off the EXE and PBDs (Jenkins calls them "artifacts").  To do this, create a Post Build step and select "Archive the Artifacts"


In the new step that is added, enter the following in for the "Files to Archive".

       **/*exe, **/*.pbd

The syntax is Apache Ant include fileset.  This particular setting tells Jenkins to find all of the EXE and PBD files in the workspace - including in any subdirectories - and copy them into the archive location.  When it does so it preserves the subdirectory structure, something we'll deal with in the next step.



Install the Copy Artifacts plugin


Now that we have the PBDs and EXE in the archive location, we want to use the Copy Artifacts Plugin to copy them to the build folder.  First we'll need to install the plugin.  Go back to the Manage Jenkins  -> Manage Plugins page, but this time select the "Available" tab.  Because there are a lot of plugins for Jenkins you might want to use the Filter option in the upper right to pear down the list.


Once again you'll need to restart the server before you can use the plugin.

Create a Copy Artifacts project


After the server has been restarted, create a new project.  We'll fire this project off after the build is complete from our original PowerBuilder project.  You might simply name the new project as the original project name with an " - Output" suffix, as I've done here.


The only thing we're going to do with this new project is create one build step.  Do that, and choose the new "Copy artifacts from another project" option.



  • Project to Copy From:  Select your original PowerBuilder project.
      
  • Artifacts to Copy:  You can leave this blank, as it defaults to copying all artifacts.
  • Target Directory:  The he location you want the files copied to, and can reference the Jenkins environmental variables using $VARIABLENAME (as opposed to %VARIABLENAME% that we used in the batch files).  For example, I'm using:

    c:\builds\$JOB_NAME\$BUILD_NUMBER

    In the previous article I was concatenating the job_name and build_number for the directory.  I like this better as I'm now putting the builds in subdirectories under the job name.
  • Flatten Directories:  The one other option you will want to change is to check the "Flatten Directories" option.  That tells the plugin to just copy over the EXE and PBD files and ignore the subdirectory structure.




Great a Post Build step to fire the Copy Artifacts project


Go back into the original PowerBuilder project.  We need to add another Post Build step. This time select "Build other projects".


And in the new post build step specify your "output" project as the project to run after the PowerBuilder project completes.



Test the new configuration


Run a build of your PowerBuilder project just to verify that it's all working.  You'll notice some differences.  In particular, the build information now shows the artifacts that were created and the "output" project as a "Downstream Project".


The build information from the "output" project isn't particularly interesting, but we can check out build folder and see that the build files are being created and stored correctly.



Caveats


This all works great.  The only issue is that (currently) there are a number of options available in the PowerBuilder target that aren't supported by the command line arguments for the PowerBuilder autocompiler.  Well we have some other options.  One is to user PowerGen, and the other is to use the command line options of the PowerBuilder IDE itself.  Fortunately, I also created a plugin for each of those as well.

PowerGen plugin


If you'd like to use PowerGen instead, grab a copy of the PowerGen plugin.  You'll need the powergencompile.hpi.  Install it as you did the other two plugins above.  Similarly, go to the Global Tool Configuration and create a PowergenCompile installation.



Like the other two plugins, the only thing you need to do here is give the installation a name and provide a path to the PowerGen executable (Pwrgn17.exe).

Go back into your project, delete the PBCompile step and add instead a PowerGen step.


And in the form that appears there selection the install you just created and the relative location of the PowerGen project file you've created for the project.


You'll need to save or copy that PowerGen project file in the workspace.

Caveats


There's a couple of caveats with this approach as well though, both of which I've shared with the folks who make PowerGen.

  • PowerGen doesn't output to the console.  The other utilities we've used so far do, and it's useful for following the progress of a build.  Instead, PowerGen (if the project file is configured for it) outputs to a log file.  What I've done is added *.log to the artifacts.  As a result, the log file PowerGen generates appears in the project summary page and I can click on it to display it after the build.



  • Powergen doesn't indicate build errors by returning a non-zero exit code.  Instead, when there are errors it writes out a pgerror.log file.  However, a non-zero exit code (also often referred to as an ERRORLEVEL code) is what Jenkins is looking for from such tools to indicate when their task has failed.  It's certainly possible that from within the plugin I could look for the existence of a pgerror.log and flag the task as failing.  However, in the long run I think it would be cleaner if PowerGen could update its behavior to include the non-zero exit code.
Note that PowerGen also has the capability to generate PBLs from source code like OrcaScript does.  I haven't looking into making use of that capability as I already have OrcaScript doing that for me.

PowerBuilder IDE Plugin


The last option is to use the PowerBuilder IDE itself to do the compile using command line arguments.  To do that, grab a copy of the PowerBuilder IDE plugin.  You'll need the pbidecompile.hpi file.  Install it as you did the others, and then go into Global Tool Configuration and give it a name and the location of the PowerBuilder IDE.


Go back and replace that last build step with the new "Build a PowerBuilder target with the PowerBuilder IDE command line" option that appears.


And in the form that appears, select the version of the install you just created and indicate the relative location of both the workspace and project file.


Note that the workspace file is a new requirement, and something that isn't part of the source pulled over from the source control server.  You'll have to copy or create a workspace file in the Jenkins workspace in order to use the PowerBuilder IDE command line approach.

Caveats


  • You have to purchase an additional license of the PowerBuilder IDE and install it on the build machine for this option.
  • Like PowerGen, the PowerBuilder IDE doesn't output to the console.  So, like PowerGen, we write the output from the IDE to a log file.
  • You need to copy or create a PBW file for the workspace, as the PBW file isn't one of the source controlled objects.
  • It appears that problems generating a build don't cause the IDE to exit with an error code.  In fact, it seems to prevent the IDE from exiting entirely.  Instead, the build appears to hang as the IDE waits for user interaction.







12 comments:

Tareq H. said...

Hello Bruce, after installing the two hpi files in Jenkins and going into "Manage Jenkins" and then "Global Tool Configuration" there is no "Orcascript" section. Just a PBCompile Section.

Going to "Manage Jenkins" option, and then the "Manage Plugins" and select the installed tab, the orcascipt pugin is installed and selected.

Installed Version is Jenkins ver. 2.110

What could be the problem?

Bruce Armstrong said...

It could be I broke something with the 0.2 release. Have you tried the 0.1 release?

Tareq H. said...

yea, i did now and your release 0.2 is broken. with 0.1 there is is a orcacript section.
But i still have a problem. After reconfigure the Buildsteps with your new plugins and the new project to safe the artefakts, thebuild process is like waiting in a endless lop on the step:

...
[projectname] $ cmd.exe /C " chcp 1252 && "C:\Program Files (x86)\Appeon\AutoCompile 17.0\orcascr170.exe" C:\Jenkins\workspace\projectname\orcascript.orca " && exit %%ERRORLEVEL%%
Aktive Codepage: 1252.
*LOADING ICON*

What could be the problem?

Tareq H. said...

ok..its not a endless loop, but that takes waaaay to long. Our Deployprocess takes with the Powerbuilder IDE Compiler 4Minutes, with Jenkins CI and the standalone compiler 21Minutes(!!!) with your new solution, the plugins 35Minutes+...

...21Minutes is way to long for us. Is there a possibility we can reach the 4 minutes with the standalone compiler and jenkins as we have with the powerbuilder IDE integrated compiler?

Tareq H. said...

me again...sorry for the spam.

Trying now the compile via the PowerBuilder IDE Plugin. Installed the ide, placed the settings, started the build session....after 1 hour still in an endless loop:

...
I0309 14:13:52.076267 736 GlobaAutoLock.cpp:60] CGlobaAutoLock Lock Successed, Thread Id: 736hFile:000003AC
I0309 14:13:52.076267 736 licenseinfomanager.cpp:1289] ProcessLock result: 1
I0309 14:13:52.076267 736 licensemanager.cpp:112] Request size :0
I0309 14:13:52.076267 736 LicenseConfigManage.cpp:155] Read Used-license-config file path:C:\ProgramData\AppeonConfig\PowerBuilder170Used.cfg
*LOADING ICON*


Bruce Armstrong said...

>>but that takes waaaay to long. Our Deployprocess takes with the Powerbuilder IDE
>>Compiler 4Minutes, with Jenkins CI and the standalone compiler 21Minutes(!!!)
>>with your new solution, the plugins 35Minutes+

Then I'm not sure what you're trying to achieve. When you save items in the PowerBuilder IDE it compiles them then and stores the compiled code in the PBL. That's why PowerBuilder doesn't need to do another compile when you run the app.

When you do a "compile" from PowerBuilder IDE or the command line utility on the PBLs you aren't really compiling anything unless you specific a full regen, you're just copying the already compiled code from the PBLs to the PBDs.

The point of using Jenkins is to do a complete build from raw source code. That means we have to import the objects into the PBLs first, which can require a couple of passes because of dependencies. That's the long pole here, something that you don't have to do from the IDE or command line utility. It will never be anywhere near as fast as compiling with a set of existing PBLs. But for something that is supposed to run automated that shouldn't be an issue.





Bruce Armstrong said...

>>Trying now the compile via the PowerBuilder IDE Plugin. Installed the ide, placed the
>>settings, started the build session....after 1 hour still in an endless loop:

I had a number of issues with the PowerBuilder IDE, as I indicated in the article. Make sure that you have a workspace file. Make sure the target file is writable. Make sure that the PowerBuilder IDE is configured to not require a database connection to compile scripts, or that it has a database it can connect to in order to satisfy that requirement. Make sure that the PowerBuilder IDE you're using (looks like 2017 in this case) has a valid license.

The IDE may fail for any number of reasons to compile the app, and it generally doesn't let you know what the problem is. What usually happens is that the IDE throws up a message box with the error information, and because we're running in command line mode under another process we never see it. You might try just running the IDE in command line mode itself on the Jenkins machine to see what error message you get and address the problem there.

Tareq H. said...

>> You might try just running the IDE in command line mode itself on the Jenkins machine to see what error message you get and address the problem there.

Could you write me in 1-2 phrases how to do that?

BTW: Thanks for your feedback! :)

Bruce Armstrong said...

Create a batch file along the lines of:

"C:\Program Files (x86)\Appeon\PowerBuilder 17.0\pb170.exe" /t pfc_ci\pfc_ci.pbt /d /out pbide.log

Where the /t argument points to the pbt file for the app. You'll need the PBLs for this, because this is the part that takes place after OrcaScript, where the IDE performs the compile using the PBLS.

Run the batch file and see what error messages appear.

Tareq H. said...

Your above command works, but if I start Jenkins with the Powerbuilder Command Line plugin to build, database errors will be displayed in the log.
...
nesla.pbl (nesla) .nesla.af_angemeldet.10: Error C0042: Can not compile without valid database sign-on.
nesla.pbl (nesla) .nesla.af_angemeldet.20: Error C0042: Can not compile without valid database sign-on.
nesla.pbl (nesla) .nesla.af_angemeldet.28: Error C0042: Can not compile without valid database sign-on.
nesla.pbl (nesla) .nesla.af_angemeldet.35: Error C0042: Can not compile without valid database sign-on.
nesla.pbl (nesla) .nesla.af_angemeldet.86: Error C0042: Can not
...


 Ironically, the Powerbuilder IDE also wants to migrate my project as soon as I open the IDE and want to log in to my workspace: "This application was created in a previous version and requires migration to the currect version"


How can it be that the above call from you correctly compiles, but as soon as I do this over jenkins he shows me such errors?

If your call is made after the failed attempt on Jenkins, the IDE also wants to re-migrate.
It looks like the call to build call over the Command Line plugin in Jenkins is broken

Tareq H. said...

Same Errors if i replace the build plugin with the batch file...connection to a databse is established, i cant understand that.

Thanks again for your feedback and sorry for the spam. Is there a way to contact you via Mail? Maybe better for your comment section

Bruce Armstrong said...

The best approach would be to open an issue in Github for the affected project.