5

Mirror Eclipse p2 repositories with Tycho

 7 months ago
source link: https://www.lorenzobettini.it/2022/04/mirror-eclipse-p2-repositories-with-tycho/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

Mirror Eclipse p2 repositories with Tycho

I had previously written about mirroring Eclipse p2 repositories (see this blog tag), but I’ll show how to do that with Tycho and one of its plugins in this post.

The goal is always the same: speed up my Maven/Tycho builds that depend on target platforms and insulate me from external servers.

The source code of this example can be found here: https://github.com/LorenzoBettini/tycho-mirror-example.

I will show how to create a mirror of a few features and bundles from a few p2 repositories so that I can then resolve a target definition file against the mirror. In the POM, I will also create a version of the target definition file modified to use the local mirror (using Ant). Moreover, I will also use a Tycho goal to validate such a modified target definition file against the local mirror. The overall procedure is also automatized in the CI (GitHub Actions). This way, we are confident that we will create a mirror that can be used locally for our builds.

First of all, let’s see the target platform I want to use during my Maven/Tycho builds. The target platform definition file is taken from my project Edelta, based on Xtext.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?pde version="3.8"?>
<target name="edelta.target" sequenceNumber="13">
<locations>
<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
<unit id="org.eclipse.equinox.executable.feature.group" version="0.0.0"/>
<unit id="org.eclipse.jdt.feature.group" version="0.0.0"/>
<unit id="org.eclipse.platform.feature.group" version="0.0.0"/>
<unit id="org.eclipse.pde.feature.group" version="0.0.0"/>
<unit id="org.eclipse.sdk" version="0.0.0"/>
<unit id="org.eclipse.emf.sdk.feature.group" version="0.0.0"/>
<unit id="org.eclipse.swtbot.forms.feature.group" version="0.0.0"/>
<unit id="org.eclipse.swtbot.ide.feature.group" version="0.0.0"/>
<unit id="org.eclipse.swtbot.eclipse.feature.group" version="0.0.0"/>
<repository location="https://download.eclipse.org/releases/2022-03"/>
</location>
<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
<unit id="org.eclipse.emf.mwe2.launcher.feature.group" version="0.0.0"/>
<repository location="https://download.eclipse.org/modeling/emft/mwe/updates/releases/2.12.2/"/>
</location>
<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
<unit id="org.eclipse.xtext.sdk.feature.group" version="0.0.0"/>
<repository location="https://download.eclipse.org/modeling/tmf/xtext/updates/releases/2.26.0/"/>
</location>
<location includeAllPlatforms="false" includeConfigurePhase="true" includeMode="planner" includeSource="true" type="InstallableUnit">
<unit id="de.itemis.xtext.antlr.feature.feature.group" version="2.1.1.v201405091103"/>
<repository location="https://download.itemis.com/updates/releases/2.1.1/"/>
</location>
<location includeAllPlatforms="false" includeConfigurePhase="true" includeMode="planner" includeSource="true" type="InstallableUnit">
<unit id="org.eclipse.epsilon.picto.feature.feature.group" version="0.0.0"/>
<unit id="org.eclipse.epsilon.emf.feature.feature.group" version="0.0.0"/>
<repository location="https://download.eclipse.org/epsilon/updates/2.4/"/>
</location>
<location includeAllPlatforms="false" includeConfigurePhase="true" includeMode="planner" includeSource="true" type="InstallableUnit">
<repository location="https://download.eclipse.org/tools/orbit/downloads/2022-03/"/>
</location>
</locations>
</target>

As you see, it’s rather complex and relies on several p2 repositories. The last repository is the Orbit repository; although it does not list any installable units, that is still required to resolve dependencies of Epsilon (see the last but one location). We have to consider this when defining our mirroring strategy.

As usual, we define a few properties at the beginning of the POM for specifying the versions of the plugin and the parts of the p2 update site we will mirror from:

  <properties>
    <tycho-version>2.7.1</tycho-version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <eclipse-version>2022-03</eclipse-version>
    <xtext-version>2.26.0</xtext-version>
  </properties>

Let’s configure the Tycho plugin for mirroring (see the documentation of the plugin for all the details of the configuration):

<plugin>
  <groupId>org.eclipse.tycho.extras</groupId>
  <artifactId>tycho-p2-extras-plugin</artifactId>
  <version>${tycho-version}</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>mirror</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <source>
      <repository>
        <id>${eclipse-version}</id>
        <layout>p2</layout>
        <url>https://download.eclipse.org/releases/${eclipse-version}</url>
      </repository>
      <repository>
        <id>${xtext-version}</id>
        <layout>p2</layout>
        <url>https://download.eclipse.org/modeling/tmf/xtext/updates/releases/${xtext-version}</url>
      </repository>
      <repository>
        <id>itemis</id>
        <layout>p2</layout>
        <url>https://download.itemis.com/updates/releases/2.1.1/</url>
      </repository>
      <repository>
        <id>Epsilon</id>
        <layout>p2</layout>
        <url>https://download.eclipse.org/epsilon/updates/2.4/</url>
      </repository>
      <repository>
        <id>Orbit ${eclipse-version}</id>
        <layout>p2</layout>
        <url>https://download.eclipse.org/tools/orbit/downloads/${eclipse-version}</url>
      </repository>
    </source>
    <!-- List of IUs to mirror. If omitted, allIUs will be mirrored. -->
    <!-- Omitted IU version element means latest version of the IU -->
    <ius>
      <iu>
        <id>org.eclipse.equinox.executable.feature.group</id>
      </iu>
      <iu>
        <id>org.eclipse.sdk.feature.group</id>
      </iu>
      <iu>
        <id>org.eclipse.swtbot.eclipse.feature.group</id>
      </iu>
      <iu>
        <id>org.eclipse.swtbot.ide.feature.group</id>
      </iu>
      <iu>
        <id>org.eclipse.xtext.sdk.feature.group</id>
      </iu>
      <iu>
        <id>org.eclipse.emf.sdk.feature.group</id>
      </iu>
      <iu>
        <id>de.itemis.xtext.antlr.feature.feature.group</id>
      </iu>
      <iu>
        <id>org.eclipse.epsilon.picto.feature.feature.group</id>
      </iu>
      <iu>
        <id>org.eclipse.epsilon.emf.feature.feature.group</id>
      </iu>
    </ius>
    <!-- The destination directory to mirror to. -->
    <destination>${user.home}/eclipse-mirrors</destination>
    <includePacked>false</includePacked>
    <includeOptional>false</includeOptional>
    <!--
      because some features, like epsilon, require an old version of org.antlr.runtime:
      org.eclipse.epsilon.common 2.4.0.202203041826
      requires 'osgi.bundle; org.antlr.runtime [3.1.1,3.5.3)'
    -->
    <latestVersionOnly>false</latestVersionOnly>
  </configuration>
</plugin>

The mirror will be generated in the user home subdirectory “eclipse-mirrors” (<destination> tag); we also define a few other mirroring options. Note that in this example, we cannot mirror only the latest versions of bundles (<latestVersionOnly>), as detailed in the comment in the POM. We also avoid mirroring the entire contents of the update sites (it would be too much). That’s why we specify single installable units. Remember that also dependencies of the listed installable units will be mirrored, so it is enough to list the main ones. You might note differences between the installable units specified in the target platform definition and those listed in the plugin configuration. Indeed, the target platform file could also be simplified accordingly, but I just wanted to have slight differences to experiment with.

If you write the above configuration in a POM file (a <packaging>pom</packaging> will be enough), you can already build the mirror running:

mvn package

Remember that the mirroring process will take several minutes depending on your Internet connection speed since it will have to download about 500 Mb of data.

You can verify that all the specified repositories are needed to create the mirror correctly. For example, try to remove this part from the POM:

<repository>
  <id>Orbit ${eclipse-version}</id>
  <layout>p2</layout>
  <url>https://download.eclipse.org/tools/orbit/downloads/${eclipse-version}</url>
</repository>

Try to create the mirror, and you should see this warning message because some requirements of Epsilon bundles cannot be resolved:

[WARNING] Mirror tool: Problems resolving provisioning plan.:
[Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; com.atlassian.commonmark 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; com.atlassian.commonmark-gfm-strikethrough 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; com.atlassian.commonmark-gfm-tables 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; com.atlassian.commonmark-heading-anchor 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; com.atlassian.commonmark-image-attributes 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; com.atlassian.commonmark-ins 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; com.atlassian.commonmark-task-list-items 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; com.atlassian.commonmark-yaml 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; net.sourceforge.plantuml 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.picto 2.3.0.202104221823
to osgi.bundle; org.apache.commons.csv 0.0.0.;
Unable to satisfy dependency from org.eclipse.epsilon.emg.engine 2.3.0.202104221823
to osgi.bundle; org.apache.commons.math3 0.0.0.]

Those requirements are found in the Orbit p2 repository, which we have just removed for testing purposes.

Unfortunately, I found no way to make the build fail in such cases, even because it’s just a warning, not an error. I guess this is a limitation of the Eclipse mirroring mechanism. However, we will now see how to verify that the mirror contains all the needed software using another mechanism.

We create a modified version of our target definition file pointing to our local mirror. To do that, we create an Ant file (create_local_target.ant):

<?xml version="1.0" encoding="ISO-8859-1"?>
<project name="CreateLocalTargetDefinitionFile" basedir="." default="create-target-file">
<!--
Replaces all the repository location elements of a .target file with
a location pointing to a local file system directory where the
mirror has to be already created
-->
<property name="local.target" value="local.target" />
<property name="orig.target" location="./example.target" />
<property name="local.mirror.path.input" location="${user.home}/eclipse-mirrors" />
<property name="local.mirror.url.input" value="file:${local.mirror.path}" />
<macrodef name="replace_win_slashes">
<attribute name="property.to.process" />
<attribute name="output.property" />
<sequential>
<loadresource property="@{output.property}">
<string value="@{property.to.process}" />
<filterchain>
<replaceregex pattern="\\" replace="/" flags="gi" />
</filterchain>
</loadresource>
<echo message="property.to.process: @{property.to.process}" />
<echo message="output.property    : ${@{output.property}}" />
</sequential>
</macrodef>
<replace_win_slashes property.to.process="${local.mirror.path.input}" output.property="local.mirror.path" />
<replace_win_slashes property.to.process="${local.mirror.url.input}" output.property="local.mirror.url" />
<target name="copy-target-file" description="Copy the .target definition file into the local.target">
<echo message="local.target: ${local.target}" />
<echo message="orig.target : ${orig.target}" />
<copy file="${orig.target}" tofile="${local.target}" overwrite="true" verbose="true" />
</target>
<target name="create-target-file" depends="copy-target-file" description="Creates a .target file from the original one, pointing to a local mirror">
<echo message="local.mirror.path: ${local.mirror.path}" />
<echo message="local.mirror.url : ${local.mirror.url}" />
<replaceregexp>
<regexp pattern="target name="(\S+)"(\.*)" />
<substitution expression="target name="local"\2" />
<fileset id="path.target" dir=".">
<include name="${local.target}" />
</fileset>
</replaceregexp>
<replaceregexp byline="true">
<regexp pattern="<repository location="(\S+)"(\.*)/>" />
<substitution expression="<repository location="${local.mirror.url}"\2/>" />
<fileset id="path.target" dir=".">
<include name="${local.target}" />
</fileset>
</replaceregexp>
</target>
</project>

Note that this also handles path separators in Windows correctly. The idea is to replace lines of the shape <repository location=”https://…”/> with <repository location=”file:/…/eclipse-mirrors”/>. This file assumes the original target file is example.target, and the modified file is generated into local.target.

Let’s call this Ant script from the POM:

<plugin>
  <artifactId>maven-antrun-plugin</artifactId>
  <!-- create the local.target -->
  <executions>
    <execution>
      <id>create-target-file</id>
      <phase>process-resources</phase>
      <configuration>
        <target>
          <ant antfile="create_local_target.ant"
                target="create-target-file">
          </ant>
        </target>
      </configuration>
      <goals>
        <goal>run</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Finally, let’s use Tycho to validate the local.target file (see the documentation of the goal):

<plugin>
  <groupId>org.eclipse.tycho.extras</groupId>
  <artifactId>target-platform-validation-plugin</artifactId>
  <version>${tycho-version}</version>
  <executions>
    <!-- Binds by default to the lifecycle phase: validate. -->
    <execution>
      <id>validate-target-platform</id>
      <goals>
        <goal>validate-target-platform</goal>
      </goals>
      <configuration>
        <targetFiles>
          <targetFile>local.target</targetFile>
        </targetFiles>
        <checkDependencies>true</checkDependencies>
        <checkProvisioning>true</checkProvisioning>
      </configuration>
    </execution>
  </executions>
</plugin>

Now, if we run:

mvn package

we build the mirror, and we create the local.target file.

Then, we can run the above goal explicitly to verify everything:

mvn target-platform-validation:[email protected]

If this goal also succeeds, we managed to create a local mirror that we can use in our local builds. Of course, in the parent POM of your project, you must configure the build so that you can switch to local.target instead of using your standard .target file. (You might want to look at the parent POM of my Edelta project to take some inspiration.)

Since we should not trust a test that we never saw failing (see also my TDD book 🙂 let’s try to verify with the incomplete mirror that we learned to create by removing the Orbit URL. We should see that our local target platform cannot be validated:

[INFO] Validating /home/bettini/work/eclipse/tycho/tycho-mirrors/tycho-mirrors/local.target...
[ERROR] Cannot resolve target definition:
[ERROR] Software being installed: org.eclipse.epsilon.emf.feature.feature.group ...
[ERROR] Missing requirement: org.eclipse.epsilon.emg.engine ...
requires 'osgi.bundle; org.apache.commons.math3 0.0.0' but it could not be found
[ERROR] Cannot satisfy dependency: org.eclipse.epsilon.core.feature.feature.group ...
depends on: org.eclipse.equinox.p2.iu; org.eclipse.epsilon.emg.engine ...
[ERROR] Cannot satisfy dependency: org.eclipse.epsilon.emf.feature.feature.group ...
depends on: org.eclipse.equinox.p2.iu; org.eclipse.epsilon.core.feature.feature.group ...

Alternatively, let’s try to build our mirror with <latestVersionOnly>true</latestVersionOnly>, and during the validation of the target platform, we get:

Error: Cannot resolve target definition:
Error: Software being installed: org.eclipse.epsilon.picto.feature.feature.group 2.4.0.202203041826
Error: Missing requirement:
org.eclipse.epsilon.common 2.4.0.202203041826 requires
'osgi.bundle; org.antlr.runtime [3.1.1,3.5.3)' but it could not be found

In fact, we mirror only the latest version of org.antlr.runtime (4.7.2.v20200218-0804), which does not satisfy that requirement. That’s why we must use with <latestVersionOnly>false</latestVersionOnly> in this example.

For completeness, this is the full POM:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>io.lorenzobettini</groupId>
  <artifactId>tycho-mirrors</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <packaging>pom</packaging>
  <properties>
    <tycho-version>2.7.1</tycho-version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <eclipse-version>2022-03</eclipse-version>
    <xtext-version>2.26.0</xtext-version>
  </properties>
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-scm-plugin</artifactId>
          <version>1.11.2</version>
        </plugin>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-resources-plugin</artifactId>
          <version>3.2.0</version>
        </plugin>
        <plugin>
          <groupId>org.codehaus.mojo</groupId>
          <artifactId>exec-maven-plugin</artifactId>
          <version>3.0.0</version>
        </plugin>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-antrun-plugin</artifactId>
          <version>3.0.0</version>
        </plugin>
        <plugin>
          <groupId>org.codehaus.mojo</groupId>
          <artifactId>build-helper-maven-plugin</artifactId>
          <version>3.2.0</version>
        </plugin>
        <plugin>
          <groupId>org.eclipse.tycho.extras</groupId>
          <artifactId>tycho-eclipserun-plugin</artifactId>
          <version>${tycho-version}</version>
        </plugin>
        <plugin>
          <groupId>org.eclipse.tycho</groupId>
          <artifactId>tycho-p2-repository-plugin</artifactId>
          <version>${tycho-version}</version>
        </plugin>
        <!-- Must be called explicitly in a separate invocation
          target-platform-validation:[email protected]
          see also https://github.com/eclipse/tycho/issues/350 -->
        <plugin>
          <groupId>org.eclipse.tycho.extras</groupId>
          <artifactId>target-platform-validation-plugin</artifactId>
          <version>${tycho-version}</version>
          <executions>
            <!-- Binds by default to the lifecycle phase: validate. -->
            <execution>
              <id>validate-target-platform</id>
              <goals>
                <goal>validate-target-platform</goal>
              </goals>
              <configuration>
                <targetFiles>
                  <targetFile>local.target</targetFile>
                </targetFiles>
                <checkDependencies>true</checkDependencies>
                <checkProvisioning>true</checkProvisioning>
              </configuration>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </pluginManagement>
    <plugins>
      <plugin>
        <artifactId>maven-antrun-plugin</artifactId>
        <!-- create the local.target -->
        <executions>
          <execution>
            <id>create-target-file</id>
            <phase>process-resources</phase>
            <configuration>
              <target>
                <ant antfile="create_local_target.ant"
                      target="create-target-file">
                </ant>
              </target>
            </configuration>
            <goals>
              <goal>run</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.eclipse.tycho.extras</groupId>
        <artifactId>tycho-p2-extras-plugin</artifactId>
        <version>${tycho-version}</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>mirror</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <source>
            <repository>
              <id>${eclipse-version}</id>
              <layout>p2</layout>
              <url>https://download.eclipse.org/releases/${eclipse-version}</url>
            </repository>
            <repository>
              <id>${xtext-version}</id>
              <layout>p2</layout>
              <url>https://download.eclipse.org/modeling/tmf/xtext/updates/releases/${xtext-version}</url>
            </repository>
            <repository>
              <id>itemis</id>
              <layout>p2</layout>
              <url>https://download.itemis.com/updates/releases/2.1.1/</url>
            </repository>
            <repository>
              <id>Epsilon</id>
              <layout>p2</layout>
              <url>https://download.eclipse.org/epsilon/updates/2.4/</url>
            </repository>
            <repository>
              <id>Orbit ${eclipse-version}</id>
              <layout>p2</layout>
              <url>https://download.eclipse.org/tools/orbit/downloads/${eclipse-version}</url>
            </repository>
          </source>
          <!-- List of IUs to mirror. If omitted, allIUs will be mirrored. -->
          <!-- Omitted IU version element means latest version of the IU -->
          <ius>
            <iu>
              <id>org.eclipse.equinox.executable.feature.group</id>
            </iu>
            <iu>
              <id>org.eclipse.sdk.feature.group</id>
            </iu>
            <iu>
              <id>org.eclipse.swtbot.eclipse.feature.group</id>
            </iu>
            <iu>
              <id>org.eclipse.swtbot.ide.feature.group</id>
            </iu>
            <iu>
              <id>org.eclipse.xtext.sdk.feature.group</id>
            </iu>
            <iu>
              <id>org.eclipse.emf.sdk.feature.group</id>
            </iu>
            <iu>
              <id>de.itemis.xtext.antlr.feature.feature.group</id>
            </iu>
            <iu>
              <id>org.eclipse.epsilon.picto.feature.feature.group</id>
            </iu>
            <iu>
              <id>org.eclipse.epsilon.emf.feature.feature.group</id>
            </iu>
          </ius>
          <!-- The destination directory to mirror to. -->
          <destination>${user.home}/eclipse-mirrors</destination>
          <includePacked>false</includePacked>
          <includeOptional>false</includeOptional>
          <!--
            because some features, like epsilon, require an old version of org.antlr.runtime:
            org.eclipse.epsilon.common 2.4.0.202203041826
            requires 'osgi.bundle; org.antlr.runtime [3.1.1,3.5.3)'
          -->
          <latestVersionOnly>false</latestVersionOnly>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

And this is the YAML file to build and verify in GitHub Actions:

name: Mirror and Verify
  push:
    paths-ignore:
      - 'README.md'
  pull_request:
    paths-ignore:
      - 'README.md'
jobs:
  build:
    strategy:
      matrix:
        os: ['ubuntu-latest', 'macos-latest', 'windows-latest']
      fail-fast: false
    runs-on: ${{ matrix.os }}
    steps:
    - uses: actions/[email protected]
    - name: Set up JDK 11
      uses: actions/[email protected]
      with:
        java-version: 11
    - name: Cache Maven packages
      uses: actions/[email protected]
      with:
        path: ~/.m2
        key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml', '**/*.yml', '**/*.target') }}
        restore-keys: ${{ runner.os }}-m2-
    - name: Cache Mirror
      uses: actions/[email protected]
      with:
        path: ~/eclipse-mirrors
        key: ${{ runner.os }}-p2-mirror-${{ hashFiles('**/pom.xml', '**/*.target') }}
        # the key match must be perfect:
        # if we change the mirror or the tp we invalidate the cache
    - name: Build the Mirror
      run: mvn package
      working-directory: tycho-mirrors
    - name: Verify the TP against the Mirror
      run: mvn target-platform-validation:[email protected]
      working-directory: tycho-mirrors
    - name: Show output contents
      run: tree ~/eclipse-mirrors
      if: runner.os == 'Linux'

I hope you found this post valuable, and happy mirroring! 🙂

Like this:

Loading...

This entry was posted in Tutorials and tagged eclipse, mirror, p2, tycho on April 24, 2022.

Post navigation

← Multibooting with GRUB


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK