Travis CI build improvements

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Travis CI build improvements

Zakariyya Mughal
Hello,

I noticed the Travis CI configuration could use some work. I suggest
taking a look at <https://github.com/travis-perl/helpers> since it can
automatically install all the dependencies and send the coverage to
Coveralls.

We could additionally look at automatically building on Windows (using
Appveyor) and macOS (using Travis CI) as well.

I helped the PDL project improve their builds and coverage in the past
using the same tools.

Regards,
- Zakariyya Mughal
_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Peter Cock
Good idea - we've spent time doing similar work for Biopython,
although we settled on codecov.io rather than coveralls.io -
both support Perl projects:

https://github.com/codecov/example-perl

Note that Biopython's coverage figures reported here hover
at around 80% based on TravisCI Linux testing only, where
we exclude our online tests and do not have all the optional
dependencies installed:

https://codecov.io/github/biopython/biopython/

I would expect BioPerl to face similar challenges in tracking
test coverage vs optional dependencies.

Peter

On Mon, Apr 17, 2017 at 5:15 PM, Zakariyya Mughal <[hidden email]> wrote:

> Hello,
>
> I noticed the Travis CI configuration could use some work. I suggest
> taking a look at <https://github.com/travis-perl/helpers> since it can
> automatically install all the dependencies and send the coverage to
> Coveralls.
>
> We could additionally look at automatically building on Windows (using
> Appveyor) and macOS (using Travis CI) as well.
>
> I helped the PDL project improve their builds and coverage in the past
> using the same tools.
>
> Regards,
> - Zakariyya Mughal
> _______________________________________________
> Bioperl-l mailing list
> [hidden email]
> http://mailman.open-bio.org/mailman/listinfo/bioperl-l
_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Fields, Christopher J
In reply to this post by Zakariyya Mughal
More than happy to have you help out.  Can you do this through a pull request (I know these are tested via travis)?  Or would you need direct commit access?

chris

On 4/17/17, 11:15 AM, "Bioperl-l on behalf of Zakariyya Mughal" <bioperl-l-bounces+cjfields=[hidden email] on behalf of [hidden email]> wrote:

    Hello,
   
    I noticed the Travis CI configuration could use some work. I suggest
    taking a look at <https://github.com/travis-perl/helpers> since it can
    automatically install all the dependencies and send the coverage to
    Coveralls.
   
    We could additionally look at automatically building on Windows (using
    Appveyor) and macOS (using Travis CI) as well.
   
    I helped the PDL project improve their builds and coverage in the past
    using the same tools.
   
    Regards,
    - Zakariyya Mughal
    _______________________________________________
    Bioperl-l mailing list
    [hidden email]
    http://mailman.open-bio.org/mailman/listinfo/bioperl-l
   


_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Zakariyya Mughal
On 2017-04-17 at 17:02:46 +0000, Fields, Christopher J wrote:
> More than happy to have you help out.  Can you do this through a pull request (I know these are tested via travis)?  Or would you need direct commit access?

Yes, I can do it all through a pull request. I can also set it up on a
fork and demonstrate the steps needed to enable the services.

As for the tests that require the network or optional dependencies, it
is possible to enable different builds conditionally. I believe the best
approach for the ones that require networking would be to create a
special branch that enables those tests. This branch can just be rebased
off the current master branch whenever a new release is uploaded.

Regards,
- Zakariyya Mughal

>
> chris
>
> On 4/17/17, 11:15 AM, "Bioperl-l on behalf of Zakariyya Mughal" <bioperl-l-bounces+cjfields=[hidden email] on behalf of [hidden email]> wrote:
>
>     Hello,
>    
>     I noticed the Travis CI configuration could use some work. I suggest
>     taking a look at <https://github.com/travis-perl/helpers> since it can
>     automatically install all the dependencies and send the coverage to
>     Coveralls.
>    
>     We could additionally look at automatically building on Windows (using
>     Appveyor) and macOS (using Travis CI) as well.
>    
>     I helped the PDL project improve their builds and coverage in the past
>     using the same tools.
>    
>     Regards,
>     - Zakariyya Mughal
>     _______________________________________________
>     Bioperl-l mailing list
>     [hidden email]
>     http://mailman.open-bio.org/mailman/listinfo/bioperl-l
>    
>
_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Peter Cock
On Mon, Apr 17, 2017 at 6:50 PM, Zakariyya Mughal <[hidden email]> wrote:
> On 2017-04-17 at 17:02:46 +0000, Fields, Christopher J wrote:
>> More than happy to have you help out.  Can you do this through
>> a pull request (I know these are tested via travis)?  Or would you
>> need direct commit access?
>
> Yes, I can do it all through a pull request. I can also set it up on a
> fork and demonstrate the steps needed to enable the services.

That's how I tinker with Biopython's TravisCI setup (on my fork). It also
means TravisCI usage counts against the individual vs the organisation
account for their load balancing, and you can try out different settings
on other services like the coverage display easily.

> As for the tests that require the network or optional dependencies, it
> is possible to enable different builds conditionally. I believe the best
> approach for the ones that require networking would be to create a
> special branch that enables those tests. This branch can just be rebased
> off the current master branch whenever a new release is uploaded.
>
> Regards,
> - Zakariyya Mughal

That's an interesting idea about running the online tests only on
specific branches. Biopython tries to run our online tests weekly
via buildbot, and this is part of our (time consuming) release
process (final run of full test suite on the release manager's own
machines).

Peter
_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Fields, Christopher J
    On Mon, Apr 17, 2017 at 6:50 PM, Zakariyya Mughal <[hidden email]> wrote:

    > As for the tests that require the network or optional dependencies, it
    > is possible to enable different builds conditionally. I believe the best
    > approach for the ones that require networking would be to create a
    > special branch that enables those tests. This branch can just be rebased
    > off the current master branch whenever a new release is uploaded.
    >
    > Regards,
    > - Zakariyya Mughal
   
    That's an interesting idea about running the online tests only on
    specific branches. Biopython tries to run our online tests weekly
    via buildbot, and this is part of our (time consuming) release
    process (final run of full test suite on the release manager's own
    machines).
   
    Peter

We did this with the 1.6.x branch for a while, as it was diverging more and more from the master branch (this was also at a point where we started cleaving out none-core-like modules) but we wanted to keep testing it.   IIRC we can also set up travis to run with specific env. variables on or off for a particular environment, which can then be used to trigger additional optional tests (similar in respects to Dist::Zilla’s release-only or author tests).   The downside is this would be triggered every commit, which we don’t really want.  So, using a branch is a much simpler, elegant solution (test only upon merge).

chris


_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Peter Cock
On Tue, Apr 18, 2017 at 2:39 PM, Fields, Christopher J
<[hidden email]> wrote:

>     On Mon, Apr 17, 2017 at 6:50 PM, Zakariyya Mughal <[hidden email]> wrote:
> …
>     > As for the tests that require the network or optional dependencies, it
>     > is possible to enable different builds conditionally. I believe the best
>     > approach for the ones that require networking would be to create a
>     > special branch that enables those tests. This branch can just be rebased
>     > off the current master branch whenever a new release is uploaded.
>     >
>     > Regards,
>     > - Zakariyya Mughal
>
>     That's an interesting idea about running the online tests only on
>     specific branches. Biopython tries to run our online tests weekly
>     via buildbot, and this is part of our (time consuming) release
>     process (final run of full test suite on the release manager's own
>     machines).
>
>     Peter
>
> We did this with the 1.6.x branch for a while, as it was diverging more
> and more from the master branch (this was also at a point where we
> started cleaving out none-core-like modules) but we wanted to keep
> testing it.   IIRC we can also set up travis to run with specific env.
> variables on or off for a particular environment, which can then be
> used to trigger additional optional tests (similar in respects to Dist::Zilla’s
> release-only or author tests).   The downside is this would be triggered
> every commit, which we don’t really want.  So, using a branch is a
> much simpler, elegant solution (test only upon merge).
>
> chris

There is also the new cron option in TravisCI, which might be useful
here - e.g. set a weekly TravisCI job and use the environment
variable $TRAVIS_EVENT_TYPE == 'cron' as a flag to trigger
the full test mode rather than off-line only tests.

https://docs.travis-ci.com/user/cron-jobs/
https://github.com/travis-ci/beta-features/issues/1

As a downside this could make coverage plots harder to interpret
as some commits should get higher coverage than others...

Peter

_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Zakariyya Mughal
On 2017-04-18 at 15:24:25 +0100, Peter Cock wrote:

> On Tue, Apr 18, 2017 at 2:39 PM, Fields, Christopher J
> <[hidden email]> wrote:
> >     On Mon, Apr 17, 2017 at 6:50 PM, Zakariyya Mughal <[hidden email]> wrote:
> > …
> >     > As for the tests that require the network or optional dependencies, it
> >     > is possible to enable different builds conditionally. I believe the best
> >     > approach for the ones that require networking would be to create a
> >     > special branch that enables those tests. This branch can just be rebased
> >     > off the current master branch whenever a new release is uploaded.
> >     >
> >     > Regards,
> >     > - Zakariyya Mughal
> >
> >     That's an interesting idea about running the online tests only on
> >     specific branches. Biopython tries to run our online tests weekly
> >     via buildbot, and this is part of our (time consuming) release
> >     process (final run of full test suite on the release manager's own
> >     machines).
> >
> >     Peter
> >
> > We did this with the 1.6.x branch for a while, as it was diverging more
> > and more from the master branch (this was also at a point where we
> > started cleaving out none-core-like modules) but we wanted to keep
> > testing it.   IIRC we can also set up travis to run with specific env.
> > variables on or off for a particular environment, which can then be
> > used to trigger additional optional tests (similar in respects to Dist::Zilla’s
> > release-only or author tests).   The downside is this would be triggered
> > every commit, which we don’t really want.  So, using a branch is a
> > much simpler, elegant solution (test only upon merge).
> >
> > chris
>
> There is also the new cron option in TravisCI, which might be useful
> here - e.g. set a weekly TravisCI job and use the environment
> variable $TRAVIS_EVENT_TYPE == 'cron' as a flag to trigger
> the full test mode rather than off-line only tests.
>
> https://docs.travis-ci.com/user/cron-jobs/
> https://github.com/travis-ci/beta-features/issues/1
>
> As a downside this could make coverage plots harder to interpret
> as some commits should get higher coverage than others...
>
> Peter

I could give this a try. If the cron builds are only run on a single
branch, then only looking at that single branch should remain consistent
modulo any network outages.

Cheers,
- Zaki Mughal
_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Zakariyya Mughal
In reply to this post by Zakariyya Mughal
On 2017-04-17 at 11:15:57 -0500, Zakariyya Mughal wrote:
> Hello,
>
> I noticed the Travis CI configuration could use some work. I suggest
> taking a look at <https://github.com/travis-perl/helpers> since it can
> automatically install all the dependencies and send the coverage to
> Coveralls.

The first part of this has been done at <https://github.com/bioperl/bioperl-live/pull/223>.

There are more details there such as splitting running the coverage
across different jobs in a build.

Regards,
- Zakariyya Mughal

> We could additionally look at automatically building on Windows (using
> Appveyor) and macOS (using Travis CI) as well.
>
> Regards,
> - Zakariyya Mughal
_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Fields, Christopher J
In reply to this post by Zakariyya Mughal
On 4/20/17, 3:00 PM, "Zakariyya Mughal" <[hidden email]> wrote:

    On 2017-04-18 at 15:24:25 +0100, Peter Cock wrote:
    > On Tue, Apr 18, 2017 at 2:39 PM, Fields, Christopher J
    > <[hidden email]> wrote:
    > >     On Mon, Apr 17, 2017 at 6:50 PM, Zakariyya Mughal <[hidden email]> wrote:
    > > …
    > >     > As for the tests that require the network or optional dependencies, it
    > >     > is possible to enable different builds conditionally. I believe the best
    > >     > approach for the ones that require networking would be to create a
    > >     > special branch that enables those tests. This branch can just be rebased
    > >     > off the current master branch whenever a new release is uploaded.
    > >     >
    > >     > Regards,
    > >     > - Zakariyya Mughal
    > >
    > >     That's an interesting idea about running the online tests only on
    > >     specific branches. Biopython tries to run our online tests weekly
    > >     via buildbot, and this is part of our (time consuming) release
    > >     process (final run of full test suite on the release manager's own
    > >     machines).
    > >
    > >     Peter
    > >
    > > We did this with the 1.6.x branch for a while, as it was diverging more
    > > and more from the master branch (this was also at a point where we
    > > started cleaving out none-core-like modules) but we wanted to keep
    > > testing it.   IIRC we can also set up travis to run with specific env.
    > > variables on or off for a particular environment, which can then be
    > > used to trigger additional optional tests (similar in respects to Dist::Zilla’s
    > > release-only or author tests).   The downside is this would be triggered
    > > every commit, which we don’t really want.  So, using a branch is a
    > > much simpler, elegant solution (test only upon merge).
    > >
    > > chris
    >
    > There is also the new cron option in TravisCI, which might be useful
    > here - e.g. set a weekly TravisCI job and use the environment
    > variable $TRAVIS_EVENT_TYPE == 'cron' as a flag to trigger
    > the full test mode rather than off-line only tests.
    >
    > https://docs.travis-ci.com/user/cron-jobs/
    > https://github.com/travis-ci/beta-features/issues/1
    >
    > As a downside this could make coverage plots harder to interpret
    > as some commits should get higher coverage than others...
    >
    > Peter
   
    I could give this a try. If the cron builds are only run on a single
    branch, then only looking at that single branch should remain consistent
    modulo any network outages.
   
    Cheers,
    - Zaki Mughal

That works for me, and if anything it could help raise a flag with modules relying on remote services that go away (this has been a pretty common issue)

chris


_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Zakariyya Mughal
On 2017-04-20 at 21:12:20 +0000, Fields, Christopher J wrote:

> On 4/20/17, 3:00 PM, "Zakariyya Mughal" <[hidden email]> wrote:
>
>     On 2017-04-18 at 15:24:25 +0100, Peter Cock wrote:
>     > On Tue, Apr 18, 2017 at 2:39 PM, Fields, Christopher J
>     > <[hidden email]> wrote:
>     > >     On Mon, Apr 17, 2017 at 6:50 PM, Zakariyya Mughal <[hidden email]> wrote:
>     > > …
>     > >     > As for the tests that require the network or optional dependencies, it
>     > >     > is possible to enable different builds conditionally. I believe the best
>     > >     > approach for the ones that require networking would be to create a
>     > >     > special branch that enables those tests. This branch can just be rebased
>     > >     > off the current master branch whenever a new release is uploaded.
>     > >     >
>     > >     > Regards,
>     > >     > - Zakariyya Mughal
>     > >
>     > >     That's an interesting idea about running the online tests only on
>     > >     specific branches. Biopython tries to run our online tests weekly
>     > >     via buildbot, and this is part of our (time consuming) release
>     > >     process (final run of full test suite on the release manager's own
>     > >     machines).
>     > >
>     > >     Peter
>     > >
>     > > We did this with the 1.6.x branch for a while, as it was diverging more
>     > > and more from the master branch (this was also at a point where we
>     > > started cleaving out none-core-like modules) but we wanted to keep
>     > > testing it.   IIRC we can also set up travis to run with specific env.
>     > > variables on or off for a particular environment, which can then be
>     > > used to trigger additional optional tests (similar in respects to Dist::Zilla’s
>     > > release-only or author tests).   The downside is this would be triggered
>     > > every commit, which we don’t really want.  So, using a branch is a
>     > > much simpler, elegant solution (test only upon merge).
>     > >
>     > > chris
>     >
>     > There is also the new cron option in TravisCI, which might be useful
>     > here - e.g. set a weekly TravisCI job and use the environment
>     > variable $TRAVIS_EVENT_TYPE == 'cron' as a flag to trigger
>     > the full test mode rather than off-line only tests.
>     >
>     > https://docs.travis-ci.com/user/cron-jobs/
>     > https://github.com/travis-ci/beta-features/issues/1
>     >
>     > As a downside this could make coverage plots harder to interpret
>     > as some commits should get higher coverage than others...
>     >
>     > Peter
>    
>     I could give this a try. If the cron builds are only run on a single
>     branch, then only looking at that single branch should remain consistent
>     modulo any network outages.
>    
>     Cheers,
>     - Zaki Mughal
>
> That works for me, and if anything it could help raise a flag with modules relying on remote services that go away (this has been a pretty common issue)

I have opened an issue with proposed changes that will make this
possible <https://github.com/bioperl/bioperl-live/issues/224>.

The cron job detects when it is being run as a cron job on a specific
branch (`network-cron-master`) and then tests the code from the
bioperl-live `master` branch.

Cheers,
- Zaki Mughal

>
> chris
>
_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
Reply | Threaded
Open this post in threaded view
|

Re: Travis CI build improvements

Fields, Christopher J
On 4/21/17, 12:42 AM, "Zakariyya Mughal" <[hidden email]> wrote:


    >     > There is also the new cron option in TravisCI, which might be useful
    >     > here - e.g. set a weekly TravisCI job and use the environment
    >     > variable $TRAVIS_EVENT_TYPE == 'cron' as a flag to trigger
    >     > the full test mode rather than off-line only tests.
    >     >
    >     > https://docs.travis-ci.com/user/cron-jobs/
    >     > https://github.com/travis-ci/beta-features/issues/1
    >     >
    >     > As a downside this could make coverage plots harder to interpret
    >     > as some commits should get higher coverage than others...
    >     >
    >     > Peter
    >    
    >     I could give this a try. If the cron builds are only run on a single
    >     branch, then only looking at that single branch should remain consistent
    >     modulo any network outages.
    >    
    >     Cheers,
    >     - Zaki Mughal
    >
    > That works for me, and if anything it could help raise a flag with modules relying on remote services that go away (this has been a pretty common issue)
   
    I have opened an issue with proposed changes that will make this
    possible <https://github.com/bioperl/bioperl-live/issues/224>.
   
    The cron job detects when it is being run as a cron job on a specific
    branch (`network-cron-master`) and then tests the code from the
    bioperl-live `master` branch.
   
    Cheers,
    - Zaki Mughal
   
    >
    > chris
    >
   
 Merged in.  I’ll have a look at the ticket.

chris


_______________________________________________
Bioperl-l mailing list
[hidden email]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l