non relational database - Unable to delete persistent objects from Google App Engine datastore

I have 2 classes AAA and BBB, where BBB contains a field of AAA type.When I call makePersistent() on 10 AAA and 10 BBB objects, the datastore ends up with 20 AAA objects and 10 BBB objects. I understand this is normal since GAE's datastore is non-relational.However, when I try to deletePersistentAll using the following,pm.newQuery(BBB.class).deletePersistentAll();pm.newQuery(AAA.class).deletePersistentAll();All BBB objects get removed as expected, but all 20 AAA objects remain. Is there something I'm missing?...Read more

How to make group permissions work in Django-nonrel for Google App Engine

I'm trying to get role-based permissions working for django-nonrel for GAE. Out of the box, it didn't seem to work, probably because of the implicit many-to-many relationship between Users and Groups, so I found and installed Per the documentation, I added permission_backend_nonrel to INSTALLED_APPS (after djangotoolbox), and defined AUTHENTICATION_BACKENDS to the appropriate class in gets me past the earlier problem ("DatabaseError: This query is not...Read more

Index error on querying google datastore using gcloud

I am trying to use google datastore for my non GAE application.For that i have created kinds and ancestor related entities in datastore using gcloud python library. Also updated datastore index configuration for all the kinds using gcd tool via WEB-INF/datastore-indexes.xml file and its status' are serving.However i can not successfully query the index based columns either in console or using gcloud lib.Here is the query & tracebackfrom gcloud import datastoreds = datastore.Client(dataset_id='XXXXXX')query = datastore.Query(ds, kind='event'...Read more

app engine ndb - NDB JsonProperty quota impact compared to old datastore properties?

I seem to remember the old datastore properties have something like 2 datastore write operations (DWOs) per storage, and perhaps a few more depending on how they are indexed.In the old datastore, I would frequently store everything I didn't need to index in a JSON string and store that as a TextProperty to save on multiples of writes.Having gotten used to saving and working with everything in JSON straight from the datastore, when switching to the NDB for a new app, I naturally used the NDB JsonProperty. As is usual, I became paranoid about opt...Read more

What's a good strategy to move data from a SQL database to Google NDB?

Hello there fellow netizens,I have a SQL database (about 600MB big) that I want to import into my GAE app. I know that one possibility would be to simpy use Google Cloud SQL, but I'd rather have the data available in NDB to get the benefits thereof. So I'm wondering, how should I think about converting the SQL schema into a NDB schemaless structure? Should I simply set up Kinds to mirror each table? How ought I deal with foreign keys that relate different tables?Any pointers are greatly appreciated!- Lee...Read more

google cloud datastore - Monotonically increasing fields in composite indexes

Suppose I have a Datastore kind with two properties listed below and an extremely high insert rate overall (but low insertion rate for individual values of random_key):random_key - a uniformly distributed large numbertime - a monotonically increasing timestamp indicating the insertion time of an entityI'm primarily concerned with queries on the composite index (random_key ASC, time DESC) and I don't care about queries on just the time field.Problem: But according to the datastore documentation, creating this composite index requires that I not...Read more

google cloud datastore - How to store a List of @Embedded objects in GAE-JDO?

I am trying to store a List of embedded objects in GAE/JDO. I am getting a Datanucleus exception: Field "Product.shippingOptions" is of type "java.util.List" yet has been specified as embedded. DataNucleus doesn't support embedding of this type. Your embedded type must be PersistenceCapable.Here's a simplified version of my classes:@PersistenceCapable(identityType=IdentityType.APPLICATION, detachable="true")public class Product{ @PrimaryKey @Persistent(valueStrategy=IdGeneratorStrategy.IDENTITY) private Long id; @Persistent(defau...Read more

Google cloud storage options for small blobs

We have about 10 millions of small files - blobs, most of them around 1-2kB, some around 100kB and a very few above 1MB, total size 50GB. They have unique integer id - key from MySQL database. They will be always accessed by the key, never searched. Access time for one blob should be to 50ms. But we want to archive them occasionally, or do some bulk deletes or updates. I need to choose the proper storage option in the Google Cloud:Storage - natural choice for blobs. But cannot do well archive and restore large number of objects - no bulk downlo...Read more

How can I get current projectId insde Google Cloud Function?

The question is simple. I need projectId in order to establish connection insde Google Cloud Function. I found the documentation where it said, that projectId is optional parameter and will be checked from GCLOUD_PROJECT, but on deployed function it didn't work.So it is the question now how can I get the projectId env variable in order to pass it as argument for Datastore connection instance, or what should be done to not pass this id and establish connection with datastore inside Google Cloud Function? Update 1I found that I actually can get v...Read more

gcloud python - Datastore best way to store aggregated nested data for better querying

I am a newbie to datastore. using it for an non GAE application. I am approaching for a better design to my use case.Storing below nested aggregated data by flattening and storing in multiple kinds for better query support. "DateTime": "2015-10-21 12:10:50","Domain": "","Events": [ { "EventName": "visit", "EventCount": "188", "Attributes_Aggregations": [ { "Name": "color", "Value_Aggregations": [ { "Value": "red", ...Read more

New Google Cloud Console not showing TEXT field

In new Developer Console TEXT field are not shown, I can see only an empty JSON Object; instead in the old one is possible to see part of the content.Maybe I have to attach a property or an annotation to allow the visualization in the Datastore Viewer of TEXT fields?Also, if I enter in the Edit section of the Entity, I can't see any kind associated to that field, it only shows the message "Editing is not supported for this type"; so it seems that the type of the field is well recognized...Anybody knows if this is a bug or an expected behaviour ...Read more

google cloud datastore - MojoExecutionException while writing to BigQuery

I am relatively new to GCP and trying out my hand to get the feel of different services offered on Google Cloud. While trying out dataflow (Beam) with Bigquery I got an exception and I am not able to find anything out from the error displayed. Any help on this will appreciated. Context: I am trying to run a dataflow pipeline which will read the whole data from datastore and then write it to Bigquery. Codeimport;import;import more

google cloud datastore - DataFlow not acking PubSub messages

Simple gcloud dataflow pipeline:PubsubIO.readStrings().fromSubscription -> Window -> ParDo -> DatastoreIO.v1().write()When load is applied to the pubsub topic, the messages are read but not acked:Jul 25, 2017 4:20:38 PM$PubsubReader statsINFO: Pubsub projects/my-project/subscriptions/my-subscription has 1000 received messages, 950 current unread messages, 843346 current unread bytes, 970 current in-flight msgs, 28367ms oldest in-flight, 1 current in-flight checkpoints, 2 max in-flight chec...Read more