Task #14390
closed
[OPS][JENKINS] slaves are not launching anymore on gcloud
Description
Jenkins is not being able to launch new slaves to run jenkins jobs. All of them fail due to an error found. I think it is on the google api as it is returning this:
Stack trace
java.lang.IllegalStateException: zone https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-east2-c not present in [https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-east1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-east1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-east1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-northeast1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-northeast1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-northeast1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-south1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-south1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-south1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-southeast1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-southeast1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/asia-southeast1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/australia-southeast1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/australia-southeast1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/australia-southeast1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-north1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-north1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-north1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west1-d, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west2-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west2-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west2-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west3-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west3-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west3-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west4-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west4-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/europe-west4-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/northamerica-northeast1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/northamerica-northeast1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/northamerica-northeast1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/southamerica-east1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/southamerica-east1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/southamerica-east1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-central1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-central1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-central1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-central1-f, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-east1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-east1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-east1-d, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-east4-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-east4-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-east4-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-west1-a, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-west1-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-west1-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-west2-c, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-west2-b, https://www.googleapis.com/compute/v1/projects/ci-veritasgenetics-0/zones/us-west2-a]
at org.jclouds.googlecomputeengine.compute.functions.MachineTypeToHardware.apply(MachineTypeToHardware.java:55)
at org.jclouds.googlecomputeengine.compute.functions.MachineTypeToHardware.apply(MachineTypeToHardware.java:39)
at shaded.com.google.common.collect.Iterators$8.transform(Iterators.java:794)
at shaded.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at shaded.com.google.common.collect.Iterators$7.computeNext(Iterators.java:646)
at shaded.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at shaded.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at shaded.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:300)
at shaded.com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:522)
at shaded.com.google.common.collect.ImmutableSet.copyOf(ImmutableSet.java:321)
at shaded.com.google.common.collect.ImmutableSet.copyOf(ImmutableSet.java:300)
at shaded.com.google.common.collect.FluentIterable.toSet(FluentIterable.java:358)
at org.jclouds.compute.config.ComputeServiceAdapterContextModule.transformGuardingNull(ComputeServiceAdapterContextModule.java:111)
at org.jclouds.compute.config.ComputeServiceAdapterContextModule.access$000(ComputeServiceAdapterContextModule.java:52)
at org.jclouds.compute.config.ComputeServiceAdapterContextModule$1.get(ComputeServiceAdapterContextModule.java:101)
at org.jclouds.compute.config.ComputeServiceAdapterContextModule$1.get(ComputeServiceAdapterContextModule.java:98)
at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:75)
at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:57)
at shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
at shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
at shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
at shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
at shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3934)
at shaded.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
at shaded.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.get(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:150)
at org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:668)
at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.get(JCloudsSlaveTemplate.java:442)
at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.provisionSlave(JCloudsSlaveTemplate.java:339)
at jenkins.plugins.jclouds.compute.JCloudsCloud.doProvisionFromTemplate(JCloudsCloud.java:439)
at jenkins.plugins.jclouds.compute.JCloudsCloud.doProvision(JCloudsCloud.java:468)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:130)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
Caused: javax.servlet.ServletException
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:789)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:253)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:124)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:503)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
Files
Updated by Fernando Monserrat over 5 years ago
I couldn't fix the new issue with the incorrect zones in the Jclouds plugin , i think it is getting them wrong after a change in google api, it retrieves us-east-2 zone which doesnt exist.
Updated by Fernando Monserrat over 5 years ago
- File Screenshot 2018-10-23 15.38.08.png Screenshot 2018-10-23 15.38.08.png added
- File Screenshot 2018-10-23 15.39.32.png Screenshot 2018-10-23 15.39.32.png added
- File Screenshot 2018-10-23 15.39.23.png Screenshot 2018-10-23 15.39.23.png added
I solved this by installing the jenkins Google Compute Engine Plugin to manage the slave nodes. https://wiki.jenkins.io/display/JENKINS/Google+Compute+Engine+Plugin
I have configured the plugin using the project ci-veritasgenetics-0, I'm attaching screenshots with current configuration to manage both kind of images we use to run tests and builds. I created 2 profiles Tests which manages the slaves to run the tests on slaves and Build profile to manage the builds, packer builds and diagnostics. The jobs are pinned to a profile using the Label configuration.
I had to reconfigure the test jobs to adjust to the home dirs of the new images , this plugin manages them differently as we've been doing it with the jcloud plugin.
it is all green now and Developer tests, run-tests , builds and packer builds are working fine. I'll keep an eye in case i need to tweak something
Updated by Fernando Monserrat over 5 years ago
- Status changed from In Progress to Feedback
Updated by Fernando Monserrat over 5 years ago
- Status changed from Feedback to Resolved
- Remaining (hours) set to 0.0
Updated by Peter Amstutz over 4 years ago
- Estimated time set to 0.00 h
- Target version deleted (
Kanban)